Methods, systems, apparatuses, and devices for facilitating provisioning of a virtual experience

ABSTRACT

A system, including a memory in communication with a processor, the memory storing instructions that when executed by the processor cause the processor to create and store a geospatial virtual environment comprising a plurality of entities each having one or more location attributes and corresponding time attributes wherein at least one of the plurality of entities is a virtual asset and wherein at least one of the plurality of entities represents a real vehicle having a defined location within a physical space having spatial coordinates that are mapped to the virtual environment, receive an updated location of the real vehicle, map the received location of the real vehicle to the geospatial virtual environment, update the entity of the geospatial virtual environment corresponding to the real vehicle with the mapped received location and output data comprising a portion of the geospatial virtual environment to a display device adapted to display to an operator of the real vehicle a mixed reality representation of at least one virtual entity.

CROSS REFERENCE TO RELATED APPLICATIONS

The present patent application is a continuation-in-part ofPCT/US2022/027665 filed May 4, 2022, which claims the benefit of U.S.Provisional Patent Application 63/183,951, filed May 4, 2021, U.S.Provisional Patent Application 63/234,261, filed Aug. 18, 2021, U.S.Provisional Patent Application 63/234,866, filed Aug. 19, 2021, and U.S.Provisional Patent Application 63/335,977, filed Apr. 28, 2022; theentire disclosures of each of which are hereby incorporated herein byreference.

FIELD OF THE INVENTION

Generally, the present disclosure relates to the field of dataprocessing. More specifically, the present disclosure relates tomethods, systems, apparatuses, and devices for facilitating provisioningof a virtual experience.

BACKGROUND OF THE INVENTION

Display devices are used for various types of training, such as insimulators. Such display devices may display virtual reality andaugmented reality content.

However, in some situations, movement of a display device with respectto a user using the display device may alter a perception of the contentthat may be displayed. For instance, due to a movement of the displaydevice due to external forces, such as movement of display devices inflight helmets due to acceleration of aircraft, the user's perception ofthe displayed content may change, which is not desired.

Therefore, there is a need for improved methods, systems, apparatusesand devices for facilitating provisioning of a virtual experience thatmay overcome one or more of the above-mentioned problems and/orlimitations.

SUMMARY

This summary is provided to introduce a selection of concepts in asimplified form, that are further described below in the DetailedDescription. This summary is not intended to identify key features oressential features of the claimed subject matter. Nor is this summaryintended to be used to limit the claimed subject matter's scope.

In accordance with exemplary and non-limiting embodiments, a systemcomprises a memory in communication with a processor, the memory storinginstructions that when executed by the processor cause the processor tocreate and store a geospatial virtual environment comprising a pluralityof entities each having one or more location attributes andcorresponding time attributes wherein at least one of the plurality ofentities is a virtual asset and wherein at least one of the plurality ofentities represents a real vehicle having a defined location within aphysical space having spatial coordinates that are mapped to the virtualenvironment, receive an updated location of the real vehicle, map thereceived location of the real vehicle to the geospatial virtualenvironment, update the entity of the geospatial virtual environmentcorresponding to the real vehicle with the mapped received location andoutput data comprising a portion of the geospatial virtual environmentto a display device adapted to display to an operator of the realvehicle a mixed reality representation of at least one virtual entity

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are incorporated in and constitute apart of this disclosure, illustrate various embodiments of the presentdisclosure. The drawings contain representations of various trademarksand copyrights owned by the Applicants. In addition, the drawings maycontain other marks owned by third parties and are being used forillustrative purposes only. All rights to various trademarks andcopyrights represented herein, except those belonging to theirrespective owners, are vested in and the property of the applicants. Theapplicants retain and reserve all rights in their trademarks andcopyrights included herein, and grant permission to reproduce thematerial only in connection with reproduction of the granted patent andfor no other purpose.

Furthermore, the drawings may contain text or captions that may explaincertain embodiments of the present disclosure. This text is included forillustrative, non-limiting, explanatory purposes of certain embodimentsdetailed in the present disclosure.

FIG. 1 is an illustration of an online platform consistent with variousembodiments of the present disclosure.

FIG. 2 shows a wearable display device for facilitating provisioning ofa virtual experience, in accordance with some embodiments.

FIG. 3 shows a wearable display device for facilitating provisioning ofa virtual experience with a compressed deformable layer, in accordancewith some embodiments.

FIG. 4 shows a wearable display device including an actuator forfacilitating provisioning of a virtual experience, in accordance withsome embodiments.

FIG. 5 shows a wearable head gear for facilitating provisioning of avirtual experience, in accordance with some embodiments.

FIG. 6 shows a method for facilitating provisioning of a virtualexperience through a wearable display device, in accordance with someembodiments.

FIG. 7 shows a method for determining a spatial parameter changeassociated with a wearable display device in relation to a user, inaccordance with some embodiments.

FIG. 8 is a block diagram of a system for facilitating provisioning of avirtual experience in accordance with some embodiments.

FIG. 9 is a block diagram of a first head mount display for facilitatingprovisioning of a virtual experience in accordance with someembodiments.

FIG. 10 is a block diagram of an apparatus for facilitating provisioningof a virtual experience in accordance with some embodiments.

FIG. 11 is a flowchart of a method of facilitating provisioning of avirtual experience in accordance with some embodiments.

FIG. 12 shows an exemplary head mount display associated with a vehiclefor facilitating provisioning of a virtual experience in accordance withsome embodiments.

FIG. 13 shows a system for facilitating provisioning of a virtualexperience, in accordance with some embodiments.

FIG. 14 shows a corrected augmented reality view, in accordance withsome embodiments.

FIG. 15 shows a chart related to the United States airspace system'sclassification scheme

FIG. 16 shows an augmented reality view shown to a real pilot while acivilian aircraft is taxiing at an airport, in accordance with anexemplary embodiment.

FIG. 17 is a block diagram of a computing device for implementing themethods disclosed herein, in accordance with some embodiments.

FIG. 18 is an illustration of an exemplary and non-limiting embodimentof a situation with assets in various positions.

FIG. 19 is an illustration of an exemplary and non-limiting embodimentof a jet cockpit.

FIG. 20 is an illustration of an exemplary and non-limiting embodimentof a pilot's helmet.

FIG. 21 is an illustration of an exemplary and non-limiting embodimentof a data distribution model.

FIG. 22 is an illustration of an exemplary and non-limiting embodimentof a flowchart of a method.

FIG. 23 is an illustration of an exemplary and non-limiting embodimentof a system for interacting with mapped data.

FIG. 24 is an illustration of an exemplary and non-limiting embodimentof a data fusion computer apparatus.

FIG. 25 is an illustration of an exemplary and non-limiting embodimentof a an application of the disclosed technology to a sports scenario.

FIG. 26 is an illustration of an exemplary and non-limiting embodimentof a training ecosystem.

FIG. 27 is an illustration of an exemplary and non-limiting embodimentof an application of the disclosed technology to a multiple viewerscenario.

FIG. 28 is an illustration of an exemplary and non-limiting embodimentof an application of the disclosed technology to a pedestrian viewerscenario.

FIG. 29 is an illustration of an exemplary and non-limiting embodimentof a user interface.

FIG. 30 is an illustration of an exemplary and non-limiting embodimentof an application of the disclosed technology to a theme park scenario.

DETAIL DESCRIPTIONS OF THE INVENTION

As a preliminary matter, it will readily be understood by one havingordinary skill in the relevant art that the present disclosure has broadutility and application. As should be understood, any embodiment mayincorporate only one or a plurality of the above-disclosed aspects ofthe disclosure and may further incorporate only one or a plurality ofthe above-disclosed features. Furthermore, any embodiment discussed andidentified as being “preferred” is considered to be part of a best modecontemplated for carrying out the embodiments of the present disclosure.Other embodiments also may be discussed for additional illustrativepurposes in providing a full and enabling disclosure. Moreover, manyembodiments, such as adaptations, variations, modifications, andequivalent arrangements, will be implicitly disclosed by the embodimentsdescribed herein and fall within the scope of the present disclosure.

Accordingly, while embodiments are described herein in detail inrelation to one or more embodiments, it is to be understood that thisdisclosure is illustrative and exemplary of the present disclosure, andare made merely for the purposes of providing a full and enablingdisclosure. The detailed disclosure herein of one or more embodiments isnot intended, nor is to be construed, to limit the scope of patentprotection afforded in any claim of a patent issuing here from, whichscope is to be defined by the claims and the equivalents thereof. It isnot intended that the scope of patent protection be defined by readinginto any claim a limitation found herein that does not explicitly appearin the claim itself.

Thus, for example, any sequence(s) and/or temporal order of steps ofvarious processes or methods that are described herein are illustrativeand not restrictive. Accordingly, it should be understood that, althoughsteps of various processes or methods may be shown and described asbeing in a sequence or temporal order, the steps of any such processesor methods are not limited to being carried out in any particularsequence or order, absent an indication otherwise. Indeed, the steps insuch processes or methods generally may be carried out in variousdifferent sequences and orders while still falling within the scope ofthe present invention. Accordingly, it is intended that the scope ofpatent protection is to be defined by the issued claim(s) rather thanthe description set forth herein.

Additionally, it is important to note that each term used herein refersto that which an ordinary artisan would understand such term to meanbased on the contextual use of such term herein. To the extent that themeaning of a term used herein—as understood by the ordinary artisanbased on the contextual use of such term— differs in any way from anyparticular dictionary definition of such term, it is intended that themeaning of the term as understood by the ordinary artisan shouldprevail.

Furthermore, it is important to note that, as used herein, “a” and “an”each generally denotes “at least one,” but does not exclude a pluralityunless the contextual use dictates otherwise. When used herein to join alist of items, “or” denotes “at least one of the items,” but does notexclude a plurality of items of the list. Finally, when used herein tojoin a list of items, “and” denotes “all of the items of the list.”

The following detailed description refers to the accompanying drawings.Wherever possible, the same reference numbers are used in the drawingsand the following description to refer to the same or similar elements.While many embodiments of the disclosure may be described,modifications, adaptations, and other implementations are possible. Forexample, substitutions, additions, or modifications may be made to theelements illustrated in the drawings, and the methods described hereinmay be modified by substituting, reordering, or adding stages to thedisclosed methods. Accordingly, the following detailed description doesnot limit the disclosure. Instead, the proper scope of the disclosure isdefined by the appended claims. The present disclosure contains headers.It should be understood that these headers are used as references andare not to be construed as limiting upon the subjected matter disclosedunder the header.

The present disclosure includes many aspects and features. Moreover,while many aspects and features relate to, and are described in thecontext of facilitating provisioning of a virtual experience,embodiments of the present disclosure are not limited to use only inthis context.

System Architecture

FIG. 1 is an illustration of an online platform 100 consistent withvarious embodiments of the present disclosure. By way of non-limitingexample, the online platform 100 to facilitate provisioning of a virtualexperience may be hosted on a centralized server 102, such as, forexample, a cloud computing service. The centralized server 102 maycommunicate with other network entities, such as, for example, anaugmented and virtual reality display device 106, a sensor system 110 ofan aircraft, database 114 (such as 3D model database) over acommunication network 104, such as, but not limited to, the Internet.Further, users of the online platform 100 may include relevant partiessuch as, but not limited to, trainees, trainers, pilots, administrators,and so on.

A user 112, such as the one or more relevant parties, may access onlineplatform 100 through a web based software application or browser. Theweb based software application may be embodied as, for example, but notbe limited to, a website, a web application, a desktop application, anda mobile application compatible with a computing device 1700.

VR and AR Helmet Disturbance Detection

FIG. 2 shows a wearable display device 200 for facilitating provisioningof a virtual experience. In some embodiments, the wearable displaydevice 200 may be utilized in conjunction with and/or to effectuateand/or facilitate operation of any element described elsewhere herein orillustrated in any figure herein. Further, the wearable display device200 may include a support member 202 configured to be mounted on a user204. Further, the support member 202 may include a structure allowingthe support member 202 to be easily mountable on the user 204. Forinstance, the wearable display device 200 may include a head mounteddevice (HMD). Further, the wearable display device 200 may include adisplay device 206 attached to the support member 202. For instance, ifthe wearable display device 200 is an HMD, the HMD may include a displaydevice in front of one eye of the user 204, (a monocular HMD), in frontof both eyes of the user 204, (a binocular HMD), an optical displaydevice (which may reflect projected images), and so on. Further, thedisplay device 206 may be configured for displaying at least one displaydata. Further, the display data may include virtual reality data relatedto a simulation, such as a training simulation. For instance, thetraining simulation may correspond to vehicular racing, such as Formula1®, and may be used by race car drivers to train for race events.Further, in an instance, the training simulation may correspond toflight training, and may be used by air force pilots for flight trainingin fighter aircraft. Further, in some embodiments, the display data mayinclude augmented reality data. Accordingly, the display data mayinclude one or more augmented reality components overlaid on top of liveimage. For instance, the augmented reality data may be related to flighttraining including a first aircraft training simultaneously with aplurality of aircrafts in different locations. Accordingly, theaugmented reality data may include augmented reality componentsdisplaying the plurality of plurality of aircrafts in differentlocations to a display device associated with a pilot of the firstaircraft. Further, the wearable display device 200 may include at leastone disturbance sensor 208 configured for sensing a disturbance in aspatial relationship between the display device 206 and the user 204.Further, the spatial relationship between the display device 206 and theuser 204 may include at least one of a distance and an orientation. Forinstance, the spatial relationship may include an exact distance, and anorientation, such as a precise angle between the display device 206 andthe eyes of the user 204.

Further, the disturbance in the spatial relationship may include achange in at least one of the distance and the orientation between thedisplay device 206 and the user 204. Further, the disturbance in thespatial relationship may lead to an alteration in how the user 204 mayview the at least one display data. For instance, if the disturbance inthe spatial relationship leads to a reduction in the distance betweenthe display device 206 and the user 204, the user 204 may perceive oneor more objects in the at least one display data to be closer. Forinstance, if the spatial relationship between the display device 206 andthe user 204 specifies a distance of “x” centimeters, and thedisturbance in the spatial relationship leads to a reduction in thedistance between the display device 206 and the user 204 to “y”centimeters, the user 204 may perceive the at least one display data tobe closer by “x-y” centimeters.

Further, the wearable display device 200 may include a processing device210 communicatively coupled with the display device 206. Further, theprocessing device 210 may be configured for receiving the at least onedisplay data. Further, the processing device 210 may be configured foranalyzing the disturbance in the spatial relationship. Further, theprocessing device 210 may be configured for generating a correction databased on the analyzing. Further, the processing device 210 may beconfigured for generating a corrected display data based on the at leastone display data and the correction data. Further, the correction datamay include an instruction to shift a perspective view of the at leastone display data to compensate for the disturbance in the spatialrelationship between the display device 206 and the user 204.Accordingly, the correction data may be generated contrary to thedisturbance in the spatial relationship.

For instance, the disturbance may include an angular disturbance,wherein the display device 206 may undergo an angular displacement as aresult of the angular disturbance. Accordingly, the correction data mayinclude an instruction of translation of the display data to compensatefor the angular disturbance. Further, the display data may be translatedalong a horizontal axis of the display data, a vertical axis of thedisplay data, a diagonal axis of the display data, and so on, to negatethe angular displacement of the display data.

Further, in an instance, the disturbance may include a longitudinaldisturbance, wherein the display device 206 may undergo a longitudinaldisplacement as a result of the longitudinal displacement. Accordingly,the correction data may include an instruction of translation of thedisplay data to compensate for the longitudinal disturbance. Further,the display data may be projected along a distance perpendicular to aline of sight of the user 204 to negate the angular displacement of thedisplay data. For instance, the display data may be projected along adistance perpendicular to the line of sight of the user 204 opposite toa direction of the longitudinal disturbance to compensate for thelongitudinal disturbance.

Further, the support member 202 may include a head gear configured to bemounted on a head of the user 204. Further, the head gear may include ahelmet configured to be worn over a crown of the head. Further, the headgear may include a shell configured to accommodate at least a part of ahead of the user 204. Further, a shape of the shell may define aconcavity to facilitate accommodation of at least the part of the head.Further, the shell may include an interior layer 212, an exterior layer214 and a deformable layer 216 disposed in between the interior layer212 and the exterior layer 214. Further, the deformable layer 216 may beconfigured to provide cushioning. Further, the display device 206 may beattached to at least one of the interior layer 212 and the exteriorlayer 214.

Further, the disturbance in the spatial relationship may be based on adeformation of the deformable layer 216 due to an acceleration of thehead gear. Further, the spatial relationship may include at least onevector representing at least one position of at least one part of thedisplay device 206 in relation to at least one eye of the user 204.Further, a vector of the at least one vector may be characterized by anorientation and a distance. For instance, the spatial relationshipbetween the display device 206 and the user 204 may include at least oneof a distance and an orientation. For instance, the spatial relationshipmay include an exact distance, and an orientation, such as a preciseangle between the display device 206 and the eyes of the user 204.Further, the spatial relationship may describe an optimal arrangement ofthe display device 206 with respect to the user 204. Further, so thatthe optimal arrangement of the display device 206 with respect to theuser 204 may allow the user to clearly view the display data withoutperceived distortion.

Further, in some embodiments, the at least one disturbance sensor 208may include an accelerometer configured for sensing the acceleration.Further, in some embodiments, the at least one disturbance sensor 208may include at least one proximity sensor configured for sensing atleast one proximity between the at least one part of the display device206 and the user 204. Further, in some embodiments, the at least onedisturbance sensor 208 may include a deformation sensor configured forsensing a deformation of the deformable layer 216.

Further, in some embodiments, the display device 206 may include asee-through display device 206 configured to allow the user 204 to viewa physical surrounding of the wearable device.

Further, in some embodiments, the at least one display data may includeat least one object model associated with at least one object. Further,in some embodiments, the generating of the corrected display data mayinclude applying at least one transformation to the at least one objectmodel based on the correction data.

Further, the applying of the at least one transformation to the at leastone object model based on the correction data may include translation ofthe display data to compensate for the angular disturbance. Forinstance, the correction data may include one or more instructions totranslate the display data along a horizontal axis of the display data,a vertical axis of the display data, a diagonal axis of the displaydata, and so on, to negate the angular displacement of the display data.Accordingly, the applying of the at least one transformation to the atleast one object model based on the correction data may includetranslation of the display data along the horizontal axis, the verticalaxis, and the diagonal axis of the display data, to negate the angulardisplacement of the display data. Further, in an instance, if thecorrection data includes an instruction of translation of the displaydata to compensate for the longitudinal disturbance, the applying of theat least one transformation to the at least one object model based onthe correction data may include translation may include projection ofthe display data along a distance perpendicular to a line of sight ofthe user 204 to negate the angular displacement of the display data. Forinstance, the applying of the at least one transform may includeprojection of the display data along a distance perpendicular to theline of sight of the user 204 opposite to a direction of thelongitudinal disturbance to compensate for the longitudinal disturbance.

Further, in some embodiments, the at least one disturbance sensor 208may include a camera configured to capture an image of each of a face ofthe user 204 and at least a part of the head gear. Further, the spatialrelationship may include disposition of at least the part of the headgear in relation to the face of the user 204.

Further, in some embodiments, the at least one disturbance sensor 208may include a camera disposed on the display device 206. Further, thecamera may be configured to capture an image of at least a part of aface of the user 204. Further, the wearable display device 200 mayinclude a calibration input device configured to receive a calibrationinput. Further, the camera may be configured to capture a referenceimage of at least the part of the face of the user 204 based onreceiving the calibration input. Further, the calibration input may bereceived in an absence of the disturbance. For instance, the calibrationinput device may include a button configured to be pushed by the user204 in absence of the disturbance whereupon the reference image of atleast the part of the face of the user 204 may be captured. Further, theanalyzing of the disturbance may include comparing the reference imagewith a current image of at least the part of the face of the user 204.Further, the current image may be captured by the camera in a presenceof the disturbance. Further, determining the correction data may includedetermining at least one spatial parameter change based on thecomparing. Further, the at least one spatial parameter change maycorrespond to at least one of a displacement of at least the part of theface relative to the camera and a rotation about at least one axis of atleast the part of the face relative to the camera.

Further, in some embodiments, the generating of the corrected displaydata may include applying at least one image transform on the at leastone display data based on the at least one spatial parameter change.

Further, in some embodiments, the wearable display device 200 mayinclude at least one actuator coupled to the display device 206 and thesupport member 202. Further, the at least one actuator may be configuredfor modifying the spatial relationship based on a correction data.

Further, the spatial relationship between the display device 206 and theuser 204 may include at least one of a distance 218 and an orientation.Further, the disturbance in the spatial relationship between the displaydevice 206 and the user 204 may include a change in at least one of thedistance 218, the angle, the direction, and the orientation. Further,the distance 218 may include a perceived distance between the user 204and the at least one display data. For instance, as shown in FIG. 3 ,the disturbance in the spatial relationship may originate due to aforward acceleration 304 of the user 204 and the wearable display device200. Accordingly, the deformation of the deformable layer 216 may leadto a disturbance in the spatial relationship leading to a change in thedistance 218 to a reduced distance 302 between the display device 206and the user 204. Accordingly, the correction data may includetransforming of the at least one display data through object levelprocessing and restoring the at least one display data to the distance218 from the user 204. Further, the object level processing may includeprojecting one or more objects in the display data at the distance 218instead of the distance 302 to oppose the disturbance in the spatialrelationship. Further, the disturbance in the spatial relationship mayinclude a change in the angle between the display device 206 and theuser 204. Further, the angle between the display device 206 and the user204 in the spatial relationship may be related to an original viewingangle related to the display data. Further, the original viewing anglerelated to the display data may be a viewing angle at which the user 204may view the display data through the display device 206. Further, thedisturbance in the spatial relationship may lead to a change in theoriginal viewing angle related to the display data. Accordingly, the atleast one display data may be transformed through pixel level processingto restore the original viewing angle related to the display data.Further, the pixel level processing may include translation of thedisplay data to compensate for the change in the angle in the spatialrelationship. Further, the display data may be translated along ahorizontal axis of the display data, a vertical axis of the displaydata, a diagonal axis of the display data, and so on, to negate theangular displacement of the display data to compensate for the change inthe angle in the spatial relationship, and to restore the originalviewing angle related to the display data.

Further, in some embodiments, the actuator may be configured formodifying the spatial relationship based on the correction data.Further, the correction data may include at least one operationalinstruction corresponding to the actuator to oppose the disturbance inthe spatial relationship, such as, but not limited to, modification ofthe distance, such as increasing of the distance 302 to the distance218. Further, the correction data may include at least one operationalinstruction corresponding to the actuator to oppose the disturbance inthe spatial relationship such as, but not limited to, the orientationopposing the disturbance in the spatial relationship.

FIG. 4 shows a wearable display device 400 for facilitating provisioningof a virtual experience, in accordance with some embodiments. In someembodiments, the wearable display device 400 may be utilized inconjunction with and/or to effectuate and/or facilitate operation of anyelement described elsewhere herein or illustrated in any figure herein.Further, the wearable display device 400 may include a support member402 configured to be mounted on a user 414. Further, the support member402 may include a deformable member 404.

Further, the wearable display device 400 may include a display device406 attached to the support member 402. Further, the display device 406may be configured for displaying at least one display data.

Further, the wearable display device 400 may include at least onedisturbance sensor 408 configured for sensing a disturbance in a spatialrelationship between the display device 406 and the support member 402.

Further, the spatial relationship between the display device 400 and theuser 414 may include at least one of a distance and an orientation. Forinstance, the spatial relationship may include an exact distance, and anorientation, such as a precise angle between the display device 406 andthe eyes of the user 414. Further, the disturbance in the spatialrelationship may include a change in the at least of the distance andthe orientation between the display device 406 and the user 414.Further, the disturbance in the spatial relationship may lead to analteration in how the user 414 may view the at least one display data.For instance, if the disturbance in the spatial relationship leads to areduction in the distance between the display device 406 and the user414, the user 414 may perceive one or more objects in the at least onedisplay data to be closer. For instance, if the spatial relationshipbetween the display device 406 and the user 414 specifies a distance of“x” centimeters, and the disturbance in the spatial relationship leadsto a reduction in the distance between the display device 406 and theuser 414 to “y” centimeters, the user 414 may perceive the at least onedisplay data to be closer by “x-y” centimeters.

Further, the wearable display device 400 may include at least oneactuator 410 coupled to the display device 406 and the support member402. Further, the at least one actuator 410 may be configured formodifying the spatial relationship between the display device 406 andthe user 414. Further, in an embodiment, the at least one actuator 410may be configured for modifying the spatial relationship to oppose thedisturbance in the spatial relationship. Further, in an embodiment, theat least one actuator 410 may be configured for modifying the spatialrelationship based on the correction data. For instance, the at leastone actuator 410 may be configured for actuating a connected motor, suchas an AC motor or a DC motor controlling an extendable rail mechanismconnecting the display device 406 and the support member 402. Forinstance, if the disturbance in the spatial relationship leads to areduction in the distance between the display device 406 and the user414, the user 414 may perceive one or more objects in the at least onedisplay data to be closer. For instance, if the spatial relationshipbetween the display device 406 and the user 414 specifies a distance of“x” centimeters, and the disturbance in the spatial relationship leadsto a reduction in the distance between the display device 406 and theuser 414 to “y” centimeters, the user 414 may perceive the at least onedisplay data to be closer by “x-y” centimeters. Accordingly, the atleast one actuator 410 may transmit an actuating signal to the connectedmotor to increase the distance between the display device 406 and theuser 414 by “x-y” centimeters to the distance of “x” centimeters.

Further, in an embodiment, the at least one actuator 410 may beconnected to a servo motor configured to control the angle in thespatial relationship through a 6-axis rotary mechanism. Accordingly, ifthe disturbance in the spatial relationship leads to a change in theangle between the display device 406 and the user 414, the user 414 mayperceive the at least one display data to be skewed. For instance, ifthe spatial relationship between the display device 406 and the user 414specifies the display device 406 to be significantly parallel to theuser 414, and the disturbance in the spatial relationship leads thedisplay device 406 to be skewed by an angle of 30 degrees towards theuser 414, the at least one actuator 410 may transmit an actuating signalto the connected servo motor, which may alter the angle in the spatialrelationship by 30 degrees oppositely to the disturbance in the spatialrelationship through the 6-axis rotary mechanism.

Further, the wearable display device 400 may include a processing device412 communicatively coupled with the display device 406. Further, theprocessing device 412 may be configured for receiving the at least onedisplay data. Further, the processing device 412 may be configured foranalyzing the disturbance in the spatial relationship. Further, theprocessing device 412 may be configured for generating the actuationdata based on the analyzing.

FIG. 5 shows a wearable display device 500 for facilitating provisioningof a virtual experience, in accordance with some embodiments. In someembodiments, the wearable display device 500 may be utilized inconjunction with and/or to effectuate and/or facilitate operation of anyelement described elsewhere herein or illustrated in any figure herein.Further, the wearable display device 500 may include a head gear 502including a shell configured to accommodate at least a part of a head ofthe user. Further, a shape of the shell may define a concavity tofacilitate accommodation of at least the part of the head. Further, theshell may include an interior layer 504, an exterior layer 506 and adeformable layer 508 disposed in between the interior layer 504 and theexterior layer 506. Further, the deformable layer 508 may be configuredto provide cushioning.

Further, the wearable display device 500 may include a display device510 attached to at least one of the interior layer 504 and the exteriorlayer 506. Further, the display device 510 may be configured fordisplaying at least one display data.

Further, the wearable display device 510 may include at least onedisturbance sensor 512 configured for sensing a disturbance in a spatialrelationship between the display device 510 and the at least one of theinterior layer 504 and the exterior layer 506.

Further, the wearable display device 500 may include a processing device514 communicatively coupled with the display device 510. Further, theprocessing device 514 may be configured for receiving the at least onedisplay data.

Further, the processing device 514 may be configured for analyzing adisturbance in the spatial relationship. Further, the processing device514 may be configured for generating a correction data based on theanalyzing. Further, the processing device 514 may be configured forgenerating a corrected display data based on the at least one displaydata and the correction data. Further, the display device 510 may beconfigured to display the corrected display data.

FIG. 6 shows a method 600 for facilitating provisioning of a virtualexperience through a wearable display device, such as the wearabledisplay device 200, in accordance with some embodiments.

At 602, the method 600 may include receiving, using a communicationdevice, a disturbance data from at least one disturbance sensor.Further, the at least one disturbance sensor may be configured forsensing a disturbance in a spatial relationship between a display deviceand a user. At 604, the method 600 may include analyzing, using aprocessing device, the disturbance in the spatial relationship. At 606,the method 600 may include generating, using the processing device, acorrection data based on the analyzing. At 608, the method 600 mayinclude generating, using the processing device, a corrected displaydata based on at least one display data and the correction data. At 610,the method 600 may include transmitting, using the communication device,the corrected display data to the wearable display device. Further, thewearable display device may be configured to be worn by the user.Further, the wearable display device may include a display device.Further, the display device may be configured for displaying thecorrected display data.

Disturbance System Calibration

FIG. 7 shows a method 700 for determining a spatial parameter change, inaccordance with some embodiments. At 702, the method 700 may includereceiving, using the communication device, a reference image of at leasta part of the face of the user. Further, the at least one disturbancesensor may include a camera disposed on the display device. Further, thecamera may be configured to capture an image of at least the part of aface of the user. Further, the wearable display device may include acalibration input device configured to receive a calibration input.Further, the camera may be configured to capture the reference image ofat least the part of the face of the user based on receiving thecalibration input. Further, the calibration input may be received in anabsence of the disturbance.

At 704, the method 700 may include receiving, using the communicationdevice, a current image of at least the part of the face of the user.Further, the current image may be captured by the camera in a presenceof the disturbance. At 706, the method 700 may include comparing, usingthe processing device, the reference image with the current image. At708, the method 700 may include determining using the processing device,at least one spatial parameter change based on the comparing. Further,the at least one spatial parameter change may correspond to at least oneof a displacement of at least the part of the face relative to thecamera and a rotation, about at least one axis, of at least the part ofthe face relative to the camera. Further, the generating of thecorrected display data may include applying at least one image transformon the at least one display data based on the at least one spatialparameter change. Further, the part of the face may include the eyes ofthe user. Further, the reference image may include at least onereference spatial parameter corresponding to the eyes. Further, thecurrent image may include at least one current spatial parametercorresponding to the eyes. Further, the at least one spatial parameterchange may be independent of a gaze of the eyes.

Presentation Data Generation

FIG. 8 is a block diagram of a system 800 for facilitating provisioningof a virtual experience in accordance with some embodiments. The system800 may include a communication device 802, a processing device 804 anda storage device 806.

The communication device 802 may be configured for receiving at leastone first sensor data corresponding to at least one first sensor 810associated with a first vehicle 808. Further, the at least one firstsensor 810 may be communicatively coupled to a first transmitter 812configured for transmitting the at least one first sensor data over afirst communication channel. In some embodiments, the first vehicle 808may be a first aircraft. Further, the first user may be a first pilot.

Further, the communication device 802 may be configured for receiving atleast one second sensor data corresponding to at least one second sensor820 associated with a second vehicle 818. Further, the at least onesecond sensor 820 may be communicatively coupled to a second transmitter822 configured for transmitting the at least one second sensor data overa second communication channel. In some embodiments, the second vehicle818 may be a second aircraft. Further, the second user may be a secondpilot.

In some embodiments, the at least one first sensor data may be receivedfrom a first On-Board-Diagnostics (OBD) system of the first vehicle 808,the at least one second sensor data may be received from a secondOn-Board-Diagnostics (OBD) system of the second vehicle 818.

Further, the communication device 802 may be configured for receiving atleast one first presentation sensor data from at least one firstpresentation sensor 828 associated with the first vehicle 808. Further,the at least one first presentation sensor 828 may be communicativelycoupled to the first transmitter configured for transmitting the atleast one first presentation sensor data over the first communicationchannel. Further, in an embodiment, the at least one first presentationsensor 828 may include a disturbance sensor, such as the disturbancesensor 208 configured for sensing a disturbance in a first spatialrelationship between at least one first presentation device 814associated with the first vehicle 808, and the first user. Further, thespatial relationship between the at least one first presentation device814 and the first user may include at least one of a distance and anorientation. For instance, the first spatial relationship may include anexact distance, and an orientation, such as a precise angle between theat least one first presentation device 814 and the eyes of the firstuser. Further, the disturbance in the first spatial relationship mayinclude a change in the at least of the distance and the orientationbetween the at least one first presentation device 814 and the firstuser.

Further, the communication device 802 may be configured for receiving atleast one second presentation sensor data from at least one secondpresentation sensor 830 associated with the second vehicle 818.

Further, in an embodiment, the at least one second presentation sensor830 may include a disturbance sensor configured for sensing adisturbance in a second spatial relationship between at least one secondpresentation device 824 associated with the second vehicle 818, and thesecond user.

Further, the at least one second presentation sensor 830 may becommunicatively coupled to the first transmitter configured fortransmitting the at least one second presentation sensor data over thesecond communication channel.

Further, the communication device 802 may be configured for transmittingat least one first optimized presentation data to at least one firstpresentation device 814 associated with the first vehicle 808. Further,in an embodiment, at least one first presentation device 814 may includea wearable display device facilitating provisioning of a virtualexperience, such as the wearable display device 200. Further, in anembodiment, the at least one first optimized presentation data mayinclude a first corrected display data generated based on a firstcorrection data.

Further, the at least one first presentation device 814 may include afirst receiver 816 configured for receiving the at least one firstoptimized presentation data over the first communication channel.Further, the at least one first presentation device 814 may beconfigured for presenting the at least one first optimized presentationdata.

Further, the communication device 802 may be configured for transmittingat least one second optimized presentation data to at least one firstpresentation device 814 associated with the first vehicle 808. Further,the first receiver 816 may be configured for receiving the at least onesecond optimized presentation data over the first communication channel.Further, the at least one first presentation device 814 may beconfigured for presenting the at least one second optimized presentationdata.

Further, in an embodiment, the at least one second optimizedpresentation data may include a second corrected display data generatedbased on a second correction data.

Further, the communication device 802 may be configured for transmittingat least one second optimized presentation data to at least one secondpresentation device 824 associated with the second vehicle 818. Further,the at least one second presentation device 824 may include a secondreceiver 826 configured for receiving the at least one second optimizedpresentation data over the second communication channel. Further, the atleast one first presentation device 824 may be configured for presentingthe at least one second optimized presentation data.

Further, the processing device 804 may be configured for analyzing theat least one first presentation sensor data associated with the firstvehicle 808.

Further, the processing device 804 may be configured for analyzing theat least one second presentation sensor data associated with the secondvehicle 818.

Further, the processing device 804 may be configured for generating thefirst correction data based on the analyzing the at least one firstpresentation sensor data associated with the first vehicle 808. Further,the first correction data may include an instruction to shift aperspective view of the at least one first optimized presentation datato compensate for the disturbance in the first spatial relationshipbetween the first presentation device 814 and the first user.Accordingly, the first correction data may be generated contrary to thedisturbance in the first spatial relationship. For instance, thedisturbance may include an angular disturbance, wherein the firstpresentation device 814 may undergo an angular displacement as a resultof the angular disturbance. Accordingly, the first correction data mayinclude an instruction of translation to generate the first correcteddisplay data included in the first optimized presentation data tocompensate for the angular disturbance.

Further, the processing device 804 may be configured for generating thesecond correction data based on the analyzing the at least one secondpresentation sensor data associated with the second vehicle 818.Further, the second correction data may include an instruction to shifta perspective view of the at least one second optimized presentationdata to compensate for the disturbance in the second spatialrelationship between the second presentation device 824 and the seconduser. Accordingly, the second correction data may be generated contraryto the disturbance in the second spatial relationship. For instance, thedisturbance may include an angular disturbance, wherein the secondpresentation device 824 may undergo an angular displacement as a resultof the angular disturbance. Accordingly, the second correction data mayinclude an instruction of translation to generate the second correcteddisplay data included in the second optimized presentation data tocompensate for the angular disturbance.

Further, the processing device 804 may be configured for generating theat least one first optimized presentation data based on the at least onesecond sensor data.

Further, the processing device 804 may be configured for generating theat least one first optimized presentation data based on the at least onefirst presentation sensor data.

Further, the processing device 804 may be configured for generating theat least one second optimized presentation data based on the at leastone first sensor data.

Further, the processing device 804 may be configured for generating theat least one second optimized presentation data based on the at leastone second presentation sensor data.

Further, the storage device 806 may be configured for storing each ofthe at least one first optimized presentation data and the at least onesecond optimized presentation data.

In some embodiments, the at least one first sensor 810 may include oneor more of a first orientation sensor, a first motion sensor, a firstaccelerometer, a first location sensor, a first speed sensor, a firstvibration sensor, a first temperature sensor, a first light sensor and afirst sound sensor. Further, the at least one second sensor 820 mayinclude one or more of a second orientation sensor, a second motionsensor, a second accelerometer, a second location sensor, a second speedsensor, a second vibration sensor, a second temperature sensor, a secondlight sensor and a second sound sensor.

In some embodiments, the at least one first sensor 810 may be configuredfor sensing at least one first physical variable associated with thefirst vehicle 808. Further, the at least one second sensor 820 may beconfigured for sensing at least one second physical variable associatedwith the second vehicle 818. In further embodiments, the at least onefirst physical variable may include one or more of a first orientation,a first motion, a first acceleration, a first location, a first speed, afirst vibration, a first temperature, a first light intensity and afirst sound. Further, the at least one second physical variable mayinclude one or more of a second orientation, a second motion, a secondacceleration, a second location, a second speed, a second vibration, asecond temperature, a second light intensity and a second sound.

In some embodiments, the at least one first sensor 810 may include afirst environmental sensor configured for sensing a first environmentalvariable associated with the first vehicle 808. Further, the at leastone second sensor 820 may include a second environmental sensorconfigured for sensing a second environmental variable associated withthe second vehicle 818.

In some embodiments, the at least one first sensor 810 may include afirst user sensor configured for sensing a first user variableassociated with a first user of the first vehicle 808. Further, the atleast one second sensor 820 may include a second user sensor configuredfor sensing a second user variable associated with a second user of thesecond vehicle 818.

In further embodiments, the first user variable may include a first userlocation and a first user orientation. Further, the second user variablemay include a second user location and a second user orientation.Further, the first presentation device may include a first head mountdisplay. Further, the second presentation device may include a secondhead mount display.

In further embodiments, the first head mount display may include a firstuser location sensor of the at least one first sensor 810 configured forsensing the first user location and a first user orientation sensor ofthe at least one first sensor 810 configured for sensing the first userorientation. The first head mount display is explained in further detailin conjunction with FIG. 9 below. Further, the second head mount displaymay include a second user location sensor of the at least one secondsensor 820 configured for sensing the second user location, a seconduser orientation sensor of the at least one second sensor 820 configuredfor sensing the second user orientation.

In further embodiments, the first vehicle 808 may include a first userlocation sensor of the at least one first sensor 810 configured forsensing the first user location and a first user orientation sensor ofthe at least one first sensor 810 configured for sensing the first userorientation. Further, the second vehicle 818 may include a second userlocation sensor of the at least one second sensor 820 configured forsensing the second user location, a second user orientation sensor ofthe at least one second sensor 820 configured for sensing the seconduser orientation.

In further embodiments, the first user orientation sensor may include afirst gaze sensor configured for sensing a first eye gaze of the firstuser. Further, the second user orientation sensor may include a secondgaze sensor configured for sensing a second eye gaze of the second user.

In further embodiments, the first user location sensor may include afirst proximity sensor configured for sensing the first user location inrelation to the at least one first presentation device 814. Further, thesecond user location sensor may include a second proximity sensorconfigured for sensing the second user location in relation to the atleast one second presentation device 824.

Further, in some embodiments, the at least one first presentation sensor828 may include at least one sensor configured for sensing at least onefirst physical variable associated with the first presentation device814 associated with the first vehicle 808, such as due to a G-Force, africtional force, and an uneven movement of the first vehicle 808. Forinstance, the at least one first presentation sensor 828 may include atleast one camera configured to monitor a movement of the firstpresentation device 814 associated with the first vehicle 808. Further,the at least one first presentation sensor 828 may include at least oneaccelerometer sensor configured to monitor an uneven movement of thefirst presentation device 814 associated with the first vehicle 808,such as due to a G-Force, a frictional force, and an uneven movement ofthe first vehicle 808. Further, the at least one first presentationsensor 828 may include at least one gyroscope sensor configured tomonitor an uneven orientation of the first presentation device 814associated with the first vehicle 808, such as due to a G-Force, africtional force, and an uneven movement of the first vehicle 808.

Further, the at least one second presentation sensor 830 may include atleast one sensor configured for sensing at least one first physicalvariable associated with the second presentation device 824 associatedwith the second vehicle 818, such as due to a G-Force, a frictionalforce, and an uneven movement of the second vehicle 818. For instance,the at least one second presentation sensor 830 may include at least onecamera configured to monitor a movement of the second presentationdevice 824 associated with the second vehicle 818. Further, the at leastone second presentation sensor 830 may include at least oneaccelerometer sensor configured to monitor an uneven movement of thesecond presentation device 824 associated with the second vehicle 818,such as due to a G-Force, a frictional force, and an uneven movement ofthe second vehicle 818. Further, the at least one second presentationsensor 830 may include at least one gyroscope sensor configured tomonitor an uneven orientation of the second presentation device 824associated with the second vehicle 818, such as due to a G-Force, africtional force, and an uneven movement of the second vehicle 818.

In some embodiments, the first head mount display may include a firstsee-through display device. Further, the second head mount display mayinclude a second see-through display device.

In some embodiments, the first head mount display may include a firstoptical marker configured to facilitate determination of one or more ofthe first user location and the first user orientation. Further, the atleast one first sensor 810 may include a first camera configured forcapturing a first image of the first optical marker. Further, the atleast one first sensor 810 may be communicatively coupled to a firstprocessor associated with the vehicle. Further, the first processor maybe configured for determining one or more of the first user location andthe first user orientation based on analysis of the first image.Further, the second head mount display may include a second opticalmarker configured to facilitate determination of one or more of thesecond user location and the second user orientation. Further, the atleast one second sensor 820 may include a second camera configured forcapturing a second image of the second optical marker. Further, the atleast one second sensor 820 may be communicatively coupled to a secondprocessor associated with the vehicle. Further, the second processor maybe configured for determining one or more of the second user locationand the second user orientation based on analysis of the second image.

In some embodiments, the first presentation device may include a firstsee-through display device disposed in a first windshield of the firstvehicle 808. Further, the second presentation device may include asecond see-through display device disposed in a second windshield of thesecond vehicle 818.

In some embodiments, the first vehicle 808 may include a firstwatercraft, a first land vehicle, a first aircraft and a firstamphibious vehicle. Further, the second vehicle 818 may include a secondwatercraft, a second land vehicle, a second aircraft and a secondamphibious vehicle.

In some embodiments, the at least one may include one or more of a firstvisual data, a first audio data and a first haptic data. Further, the atleast one second optimized presentation data may include one or more ofa second visual data, a second audio data and a second haptic data.

In some embodiments, the at least one first presentation device 814 mayinclude at least one environmental variable actuator configured forcontrolling at least one first environmental variable associated withthe first vehicle 808 based on the first optimized presentation data.Further, the at least one second presentation device 824 may include atleast one environmental variable actuator configured for controlling atleast one second environmental variable associated with the secondvehicle 818 based on the second optimized presentation data. In furtherembodiments, the at least one first environmental variable may includeone or more of a first temperature level, a first humidity level, afirst pressure level, a first oxygen level, a first ambient light, afirst ambient sound, a first vibration level, a first turbulence, afirst motion, a first speed, a first orientation and a firstacceleration, the at least one second environmental variable may includeone or more of a second temperature level, a second humidity level, asecond pressure level, a second oxygen level, a second ambient light, asecond ambient sound, a second vibration level, a second turbulence, asecond motion, a second speed, a second orientation and a secondacceleration.

In some embodiments, the first vehicle 808 may include each of the atleast one first sensor 810 and the at least one first presentationdevice 814. Further, the second vehicle 818 may include each of the atleast one second sensor 820 and the at least one second presentationdevice 824.

In some embodiments, the storage device 806 may be further configuredfor storing a first three-dimensional model corresponding to the firstvehicle 808 and a second three-dimensional model corresponding to thesecond vehicle 818. Further, the generating of the first optimizedpresentation data may be based further on the second three-dimensionalmodel. Further, the generating of the second optimized presentation datamay be based further on the first three-dimensional model.

Further, the generating of the first optimized presentation data may bebased on the determining of the unwanted movement of the associated withthe first presentation device 814 associated with the first vehicle 808,such as due to a G-Force, a frictional force, and an uneven movement ofthe first vehicle 808. For instance, the at least one first presentationsensor 828 may include at least one camera configured to monitor amovement of the first presentation device 814 associated with the firstvehicle 808. Further, the at least one first presentation sensor 828 mayinclude at least one accelerometer sensor configured to monitor anuneven movement of the first presentation device 814 associated with thefirst vehicle 808, such as due to a G-Force, a frictional force, and anuneven movement of the first vehicle 808. Further, the at least onefirst presentation sensor 828 may include at least one gyroscope sensorconfigured to monitor an uneven orientation of the first presentationdevice 814 associated with the first vehicle 808, such as due to aG-Force, a frictional force, and an uneven movement of the first vehicle808.

Further, the generating of the second optimized presentation data may bebased on the determining of the unwanted movement of the secondpresentation device 824 associated with the second vehicle 818, such asdue to a G-Force, a frictional force, and an uneven movement of thesecond vehicle 818. For instance, the at least one second presentationsensor 830 may include at least one camera configured to monitor amovement of the second presentation device 824 associated with thesecond vehicle 818. Further, the at least one second presentation sensor830 may include at least one accelerometer sensor configured to monitoran uneven movement of the second presentation device 824 associated withthe second vehicle 818, such as due to a G-Force, a frictional force,and an uneven movement of the second vehicle 818. Further, the at leastone second presentation sensor 830 may include at least one gyroscopesensor configured to monitor an uneven orientation of the secondpresentation device 824 associated with the second vehicle 818, such asdue to a G-Force, a frictional force, and an uneven movement of thesecond vehicle 818.

In some embodiments, the communication device 802 may be furtherconfigured for receiving an administrator command from an administratordevice. Further, the generating of one or more of the first optimizedpresentation data and the second optimized presentation data may bebased further on the administrator command. In further embodiments, theat least one first presentation model may include at least one firstvirtual object model corresponding to at least one first virtual object.Further, the at least one second presentation model may include at leastone second virtual object model corresponding to at least one secondvirtual object. Further, the generating of the at least one firstvirtual object model may be independent of the at least one secondsensor model. Further, the generating of the at least one second virtualobject model may be independent of the at least one first sensor model.Further, the generating of one or more of the at least one first virtualobject model and the at least one second virtual object model may bebased on the administrator command. Further, the storage device 806 maybe configured for storing the at least one first virtual object modeland the at least one second virtual object model.

In further embodiments, the administrator command may include a virtualdistance parameter. Further, the generating of each of the at least onefirst optimized presentation data and the at least one second optimizedpresentation data may be based on the virtual distance parameter.

In further embodiments, the at least one first sensor data may includeat least one first proximity data corresponding to at least one firstexternal real object in a vicinity of the first vehicle 808. Further,the at least one second sensor data may include at least one secondproximity data corresponding to at least one second external real objectin a vicinity of the second vehicle 818. Further, the generating of theat least one first optimized presentation data may be based further onthe at least one second proximity data. Further, the generating of theat least one second optimized presentation data may be based further onthe at least one first proximity data. In further embodiments, the atleast one first external real object may include a first cloud, a firstlandscape feature, a first man-made structure and a first naturalobject. Further, the at least one second external real object mayinclude a second cloud, a second landscape feature, a second man-madestructure and a second natural object.

In some embodiments, the at least one first sensor data may include atleast one first image data corresponding to at least one first externalreal object in a vicinity of the first vehicle 808. Further, the atleast one second sensor data may include at least one second image datacorresponding to at least one second external real object in a vicinityof the second vehicle 818. Further, the generating of the at least onefirst optimized presentation data may be based further on the at leastone second image data. Further, the generating of the at least onesecond optimized presentation data may be based further on the at leastone first image data.

In some embodiments, the communication device 802 may be furtherconfigured for transmitting a server authentication data to the firstreceiver 816. Further, the first receiver 816 may be communicativelycoupled to first processor associated with the first presentationdevice. Further, the first processor may be communicatively coupled to afirst memory device configured to store a first authentication data.Further, the first processor may be configured for performing a firstserver authentication based on the first authentication data and theserver authentication data. Further, the first processor may beconfigured for controlling presentation of the at least one firstoptimized presentation data on the at least one first presentationdevice 814 based on the first server authentication. Further, thecommunication device 802 may be configured for transmitting a serverauthentication data to the second receiver 826. Further, the secondreceiver 826 may be communicatively coupled to second processorassociated with the second presentation device. Further, the secondprocessor may be communicatively coupled to a second memory deviceconfigured to store a second authentication data. Further, the secondprocessor may be configured for performing a second serverauthentication based on the second authentication data and the serverauthentication data. Further, the second processor may be configured forcontrolling presentation of the at least one second optimizedpresentation data on the at least one second presentation device 824based on the second server authentication. Further, the communicationdevice 802 may be configured for receiving a first client authenticationdata from the first transmitter 812. Further, the storage device 806 maybe configured for storing the first authentication data. Further, thecommunication device 802 may be configured for and receiving a secondclient authentication data from the second transmitter 822. Further, thestorage device 806 may be configured for storing the secondauthentication data. Further, the processing device 804 may be furtherconfigured for performing a first client authentication based on thefirst client authentication data and the first authentication data.Further, the generating of the at least one second optimizedpresentation data may be further based on the first clientauthentication. Further, the processing device 804 may be configured forperforming a second client authentication based on the second clientauthentication data and the second authentication data. Further, thegenerating of the at least one first optimized presentation data may befurther based on the second client authentication.

FIG. 9 is a block diagram of a first head mount display 900 forfacilitating provisioning of a virtual experience in accordance withsome embodiments. The first head mount display 900 may include a firstuser location sensor 902 of the at least one first sensor configured forsensing the first user location and a first user orientation sensor 904of the at least one first sensor configured for sensing the first userorientation.

Further, the first head mount display 900 may include a display device906 to present visuals. Further, in an embodiment, the display device906 may be configured for displaying the first optimized display data,as generated by the processing device 804.

Further, the first head mount display 900 may include a processingdevice 908 configured to obtain sensor data from the first user locationsensor 902 and the first user orientation sensor 904. Further, theprocessing device 908 may be configured to send visuals to the displaydevice 906.

FIG. 10 is a block diagram of an apparatus 1000 for facilitatingprovisioning of a virtual experience in accordance with someembodiments. The apparatus 1000 may include at least one first sensor1002 (such as the at least one first sensor 810) configured for sensingat least one first sensor data associated with a first vehicle (such asthe first vehicle 808).

Further, the apparatus 1000 may include at least one first presentationsensor 1010 (such as the at least one first presentation sensor 828)configured for sensing at least one first presentation sensor dataassociated with a first vehicle (such as the first vehicle 808).Further, in an embodiment, the at least one first presentation sensor1010 may include a disturbance sensor, such as the disturbance sensor208 configured for sensing a disturbance in a first spatial relationshipbetween at least one first presentation device 1008 associated with thefirst vehicle, and a first user. Further, the spatial relationshipbetween the at least one first presentation device 1008 and the firstuser may include at least one of a distance and an orientation. Forinstance, the first spatial relationship may include an exact distance,and an orientation, such as a precise angle between the at least onefirst presentation device 1008 and the eyes of the first user. Further,the disturbance in the first spatial relationship may include a changein the at least of the distance and the orientation between the at leastone first presentation device 814 and the first user.

Further, the apparatus 1000 may include a first transmitter 1004 (suchas the first transmitter 812) configured to be communicatively coupledto the at least first sensor 1002, and the at least one firstpresentation sensor 1010. Further, the first transmitter 1004 may beconfigured for transmitting the at least one first sensor data and theat least one first presentation sensor data to a communication device(such as the communication device 802) of a system over a firstcommunication channel.

Further, the apparatus 1000 may include a first receiver 1006 (such asthe first receiver 816) configured for receiving the at least one firstoptimized presentation data from the communication device over the firstcommunication channel.

Further, the apparatus 1000 may include the at least one firstpresentation device 1008 (such as the at least one first presentationdevice 814) configured to be communicatively coupled to the firstreceiver 1006. The at least one first presentation device 1008 may beconfigured for presenting the at last one first optimized presentationdata.

Further, the communication device may be configured for receiving atleast one second sensor data corresponding to at least one second sensor(such as the at least one second sensor 820) associated with a secondvehicle (such as the second vehicle 818). Further, the at least onesecond sensor may be communicatively coupled to a second transmitter(such as the second transmitter 822) configured for transmitting the atleast one second sensor data over a second communication channel.Further, the system may include a processing device (such as theprocessing device 804) communicatively coupled to the communicationdevice. Further, the processing device may be configured for generatingthe at least one first optimized presentation data based on the at leastone second sensor data.

FIG. 11 is a flowchart of a method 1100 of facilitating provisioning ofa virtual experience in accordance with some embodiments. At 1102, themethod 1100 may include receiving, using a communication device (such asthe communication device 802), at least one first sensor datacorresponding to at least one first sensor (such as the at least onefirst sensor 810) associated with a first vehicle (such as the firstvehicle 808). Further, the at least one first sensor may becommunicatively coupled to a first transmitter (such as the firsttransmitter 812) configured for transmitting the at least one firstsensor data over a first communication channel.

At 1104, the method 1100 may include receiving, using the communicationdevice, at least one second sensor data corresponding to at least onesecond sensor (such as the at least one second sensor 820) associatedwith a second vehicle (such as the second vehicle 818). Further, the atleast one second sensor may be communicatively coupled to a secondtransmitter (such as the second transmitter 822) configured fortransmitting the at least one second sensor data over a secondcommunication channel.

At 1106, the method 1100 may include receiving, using the communicationdevice, a first presentation sensor data corresponding to at least onefirst presentation sensor 828 associated with the first vehicle.Further, the at least one first presentation sensor may becommunicatively coupled to the first transmitter configured fortransmitting the at least one first presentation sensor data over thefirst communication channel. Further, the first presentation sensor mayinclude at least one sensor configured to monitor a movement of at leastone first presentation device associated with the first vehicle, such asdue to a G-Force, a frictional force, and an uneven movement of thefirst vehicle. For instance, the at least one first presentation sensormay include at least one camera configured to monitor a movement of theat least one first presentation device associated with the firstvehicle. Further, the at least one first presentation sensor may includeat least one accelerometer sensor configured to monitor an unevenmovement of the at least one first presentation device associated withthe first vehicle, such as due to a G-Force, a frictional force, and anuneven movement of the first vehicle. Further, the at least one firstpresentation sensor may include at least one gyroscope sensor configuredto monitor an uneven orientation of the at least one first presentationdevice associated with the first vehicle, such as due to a G-Force, africtional force, and an uneven movement of the first vehicle.

At 1108, the method 1100 may include receiving, using the communicationdevice, a second presentation sensor data corresponding to at least onesecond presentation sensor 830 associated with the second vehicle.Further, the at least one second presentation sensor may becommunicatively coupled to the second transmitter configured fortransmitting the at least one second presentation sensor data over thesecond communication channel. Further, the second presentation sensormay include at least one sensor configured to monitor a movement of atleast one second presentation device associated with the second vehicle,such as due to a G-Force, a frictional force, and an uneven movement ofthe second vehicle. For instance, the at least one second presentationsensor may include at least one camera configured to monitor a movementof the at least one second presentation device associated with thesecond vehicle. Further, the at least one second presentation sensor mayinclude at least one accelerometer sensor configured to monitor anuneven movement of the at least one second presentation deviceassociated with the second vehicle, such as due to a G-Force, africtional force, and an uneven movement of the second vehicle. Further,the at least one second presentation sensor may include at least onegyroscope sensor configured to monitor an uneven orientation of the atleast one second presentation device associated with the second vehicle,such as due to a G-Force, a frictional force, and an uneven movement ofthe second vehicle.

At 1110, the method 1100 may include analyzing, using a processingdevice, the at least one first sensor data and the at least one firstpresentation sensor data to generate at least one first modifiedpresentation data. The analyzing may include determining an unwantedmovement of the at least one first presentation device associated withthe first vehicle, such as due to a G-Force, a frictional force, and anuneven movement of the first vehicle. Further, the unwanted movement ofthe at least one first presentation device associated with the firstvehicle may include an upward movement, a downward movement, a leftwardmovement, and a rightward movement. Further, the generating of the atleast one first optimized presentation data may be based on the unwantedmovement of the at least one first presentation device associated withthe first vehicle, such as due to a G-Force, a frictional force, and anuneven movement of the first vehicle. For instance, the generating ofthe at least one first optimized presentation data may be based onnegating an effect of the unwanted movement of the at least one firstpresentation device associated with the first vehicle. For instance, ifthe unwanted movement of the at least one first presentation deviceassociated with the first vehicle includes an upward movement, adownward movement, a leftward movement, and a rightward movement, thegenerating of the at least one first optimized presentation data mayinclude moving one or more components of the at least one first modifiedpresentation data in an oppositely downward direction, an upwarddirection, a rightward direction, and a leftward direction respectively.

At 1112, the method 1100 may include analyzing, using a processingdevice, the at least one second sensor data and the at least one secondpresentation sensor data to generate at least one second presentationdata. The analyzing may include determining an unwanted movement of theat least one second presentation device associated with the secondvehicle, such as due to a G-Force, a frictional force, and an unevenmovement of the second vehicle. Further, the unwanted movement of the atleast one second presentation device associated with the second vehiclemay include an upward movement, a downward movement, a leftwardmovement, and a rightward movement. Further, the generating of the atleast one second optimized presentation data may be based on theunwanted movement of the at least one second presentation deviceassociated with the second vehicle, such as due to a G-Force, africtional force, and an uneven movement of the second vehicle. Forinstance, the generating of the at least one second optimizedpresentation data may be based on negating an effect of the unwantedmovement of the at least one second presentation device associated withthe second vehicle. For instance, if the unwanted movement of the atleast one second presentation device associated with the second vehicleincludes an upward movement, a downward movement, a leftward movement,and a rightward movement, the generating of the at least one secondoptimized presentation data may include moving one or more components ofthe at least one second presentation data in an oppositely downwarddirection, an upward direction, a rightward direction, and a leftwarddirection respectively.

At 1114, the method 1100 may include transmitting, using thecommunication device, at least one first optimized presentation data toat least one first presentation device associated with the firstvehicle. Further, the at least one first presentation device may includea first receiver (such as the first receiver 816) configured forreceiving the at least one first modified presentation data over thefirst communication channel. Further, the at least one presentationdevice may be configured for presenting the at least one first optimizedpresentation data.

At 1116, the method 1100 may include transmitting, using thecommunication device, at least one second optimized presentation data toat least one second presentation device (such as the at least one secondpresentation device 824) associated with the second vehicle. Further,the at least one second presentation device may include a secondreceiver (such as the second receiver 826) configured for receiving theat least one second presentation data over the second communicationchannel. Further, the at least one presentation device may be configuredfor presenting the at least one second optimized presentation data.

At 1118, the method 1100 may include storing, using a storage device(such as the storage device 806), each of the at least one firstoptimized presentation data and the at least one second optimizedpresentation data.

FIG. 12 shows an exemplary head mount display 1200 associated with avehicle (such as the first vehicle 808) for facilitating provisioning ofa virtual experience in accordance with some embodiments. Further, thevehicle may include a watercraft, a land vehicle, an aircraft and anamphibious vehicle. The head mount display 1200 associated with thevehicle may be worn by a user, such as a driver or operator of thevehicle while driving or operating the vehicle for facilitatingprovisioning of a virtual experience.

The head mount display 1200 may include a display device 1202 (such asthe display device 906) to present visuals. The display device 1202 mayinclude a first see-through display device.

Further, under motion, such as under extreme aerobatic maneuvers, suchas G loading (gravity loading) the head mount display 1200 mayexperience one or more forces. Accordingly, a structure 1204 of the headmount display 1200 may exhibit slight movement, leading to the displaydevice 1202 shifting from a desired position. For instance, thestructure 12 d 04 of the head mount display 1200 may be compressed ontothe head of a user 1208 leading to a movement of the display device1202, such as by 3-4 mm.

Further, the head mount display 1200 may include a presentation sensor1206 (such as the first presentation sensor 828) configured for sensingat least one first physical variable (such as the movement) associatedwith the head mount display 1200, such as due to a G-Force, a frictionalforce, and an uneven movement of the vehicle. For instance, thepresentation sensor 1206 may include at least one camera configured tomonitor a movement, or compression of the head mount display 1200associated with the vehicle. Further, the presentation sensor 1206 mayinclude at least one accelerometer sensor configured to monitor anuneven movement of the head mount display 1200 associated with thevehicle, such as due to a G-Force, a frictional force, and an unevenmovement of the vehicle. Further, the presentation sensor 1206 mayinclude at least one gyroscope sensor configured to monitor an unevenorientation of the head mount display 1200 associated with the vehicle,such as due to a G-Force, a frictional force, and an uneven movement ofthe vehicle.

Further, the head mount display 1200 may include a transmitter (notshown—such as the first transmitter 812) configured to becommunicatively coupled to the presentation sensor 1206. Further, thetransmitter may be configured for transmitting the presentation sensordata to a communication device (such as the communication device 802) ofa system over a communication channel.

Further, the head mount display 1200 may include a first receiver (notshown—such as the first receiver 816) configured to be communicativelycoupled to the display device 1202. Further, the first receiver may beconfigured for receiving the at least one modified presentation datafrom the communication device over the communication channel. Further,the modified presentation data may negate the slight movement of thehead mount display 1200, leading to the display device 1202 shiftingfrom the desired position.

Further, the communication device may be configured for receiving atleast one second sensor data corresponding to at least one second sensor(such as the at least one second sensor 820) associated with a secondvehicle (such as the second vehicle 818). Further, the at least onesecond sensor may be communicatively coupled to a second transmitter(such as the second transmitter 822) configured for transmitting the atleast one second sensor data over a second communication channel.Further, the system may include a processing device (such as theprocessing device 804) communicatively coupled to the communicationdevice. Further, the processing device may be configured for generatingthe presentation data based on the at least one second sensor data.

FIG. 13 shows a system 1300 for facilitating provisioning of a virtualexperience, in accordance with some embodiments. The system 1300 mayinclude a communication device 1302 configured for receiving at leastone first sensor data corresponding to at least one first sensor 1310associated with a first vehicle 1308. Further, the at least one firstsensor 1310 may be communicatively coupled to a first transmitter 1312configured for transmitting the at least one first sensor data over afirst communication channel.

Further, the communication device 1302 may be configured for receivingat least one second sensor data corresponding to at least one secondsensor 1316 associated with a second vehicle 1314. Further, the at leastone second sensor 1316 may include a second location sensor configuredto detect a second location associated with the second vehicle 1314.Further, the at least one second sensor 1316 may be communicativelycoupled to a second transmitter 1318 configured for transmitting the atleast one second sensor data over a second communication channel.Further, in some embodiments, the at least one second sensor 1316 mayinclude a second user sensor configured for sensing a second uservariable associated with a second user of the second vehicle 1314.Further, the second user variable may include a second user location anda second user orientation.

Further, in some embodiments, the at least one second sensor 1316 mayinclude a disturbance sensor, such as the disturbance sensor 208configured for sensing a disturbance in a spatial relationship between asecond presentation device 1320 associated with the second vehicle 1314and the second user of the second vehicle 1314. Further, the spatialrelationship between the second presentation device 1320 and the seconduser may include at least one of a distance and an orientation. Forinstance, the spatial relationship may include an exact distance, and anorientation, such as a precise angle between the second presentationdevice 1320 and the eyes of the second user.

Further, the disturbance in the spatial relationship may include achange in the at least of the distance and the orientation between thesecond presentation device 1320 and the second user. Further, thedisturbance in the spatial relationship may lead to an alteration in howthe second user may view at least one second presentation data. Forinstance, if the disturbance in the spatial relationship leads to areduction in the distance between the second presentation device 1320and the second user, the second user may perceive one or more objects inthe at least one second presentation data to be closer. For instance, ifthe spatial relationship between the second presentation device 1320 andthe second user specifies a distance of “x” centimeters, and thedisturbance in the spatial relationship leads to a reduction in thedistance between the second presentation device 1320 and the second userto “y” centimeters, the second user may perceive the at least one secondpresentation data to be closer by “x-y” centimeters.

Further, the communication device 1302 may be configured fortransmitting the at least one second presentation data to the at leastone second presentation device 1320 associated with the second vehicle1314. Further, the at least one second presentation data may include atleast one second virtual object model corresponding to at least onesecond virtual object. Further, in some embodiments, the at least onesecond virtual object may include one or more of a navigational markerand an air-corridor.

Further, in an embodiment, the at least one second presentation data mayinclude a second corrected display data generated based on a secondcorrection data. Further, the at least one second presentation device1320 may include a second receiver 1322 configured for receiving the atleast one second presentation data over the second communicationchannel. Further, the at least one second presentation device 1320 maybe configured for presenting the at least one second presentation data.Further, in some embodiments, the at least one second presentationdevice 1320 may include a second head mount display. Further, the secondhead mount display may include a second user location sensor of the atleast one second sensor 1316 configured for sensing the second userlocation and a second user orientation sensor of the at least one secondsensor 1316 configured for sensing the second user orientation. Further,the second head mount display may include a second see-through displaydevice.

Further, in some embodiments, the at least one second virtual objectmodel may include a corrected augmented reality view, such as thecorrected augmented reality view 1400. Further, the augmented realityview 1400 may include one or more second virtual objects such as anavigational marker 1408, and a skyway 1406 as shown in FIG. 14 ).

Further, the system 1300 may include a processing device 1304 configuredfor generating the at least one second presentation data based on the atleast one first sensor data and the at least one second sensor data.Further, the generating of the at least one second virtual object modelmay be independent of the at least one first sensor data. Further, insome embodiments, the processing device 1304 may be configured fordetermining a second airspace class (with reference to FIG. 15 )associated with the second vehicle 1314 based on the second locationincluding a second altitude associated with the second vehicle 1314.Further, the generating of the at least one second virtual object modelmay be based on the second airspace class.

Further, the processing device 1304 may be configured for generating thesecond correction data based on the analyzing the at least one secondsensor data associated with the second vehicle 1314. Further, the secondcorrection data may include an instruction to shift a perspective viewof the at least one second presentation data to compensate for thedisturbance in the spatial relationship between the second presentationdevice 1320 and the second user. Accordingly, the second correction datamay be generated contrary to the disturbance in the spatialrelationship. For instance, the disturbance may include an angulardisturbance, wherein the second presentation device 1320 may undergo anangular displacement as a result of the angular disturbance.Accordingly, the second correction data may include an instruction oftranslation to generate the second corrected display data included inthe second presentation data to compensate for the angular disturbance.

For instance, if the at least one second presentation data includes theat least one second virtual object model may include a correctedaugmented reality view, such as the corrected augmented reality view1400, the second correction data may include an instruction to shift aperspective view of the at least one second presentation data tocompensate for the disturbance in the spatial relationship between thesecond presentation device 1320 and the second user (such as a pilot1402). For instance, if the disturbance in the spatial relationshipincludes a reduction in the distance between the second presentationdevice 1320, the second correction data may include an instruction toshift a perspective view of the at least one second presentation data tocompensate for the disturbance in the spatial relationship between thesecond presentation device 1320 and the second user, such as byprojection of the one or more second virtual objects, such as thenavigational marker 1408, and the skyway 1406 at a distance tocompensate the disturbance and to generate the corrected augmentedreality view 1400\

Further, the system 1300 may include a storage device 1306 configuredfor storing the at least one second presentation data. Further, in someembodiments, the storage device 1306 may be configured for retrievingthe at least one second virtual object model based on the secondlocation associated with the second vehicle 1314. Further, in someembodiments, the storage device 1306 may be configured for storing afirst three-dimensional model corresponding to the first vehicle 1308.Further, the generating of the second presentation data may be based onthe first three-dimensional model.

Further, in some embodiments, the communication device 1302 may beconfigured for receiving an administrator command from an administratordevice. Further, the generating of the at least one second virtualobject model may be based on the administrator command.

Further, in some embodiments, the communication device 1302 may beconfigured for transmitting at least one first presentation data to atleast one first presentation device (not shown) associated with thefirst vehicle 1308. Further, the at least one first presentation devicemay include a first receiver configured for receiving the at least onefirst presentation data over the first communication channel. Further,the at least one first presentation device may be configured forpresenting the at least one first presentation data. Further, in someembodiments, the processing device 1304 may be configured for generatingthe at least one first presentation data based on the at least onesecond sensor data. Further, in some embodiments, the storage device1306 may be configured for storing the at least one first presentationdata. Further, in some embodiments, the storage device 1306 may beconfigured for storing a second three-dimensional model corresponding tothe second vehicle 1314. Further, the generating of the firstpresentation data may be based on the second three-dimensional model.

Further, in some embodiments, the at least one first presentation datamay include at least one first virtual object model corresponding to atleast one first virtual object. Further, the generating of the at leastone first virtual object model may be independent of the at least onesecond sensor data. Further, the storage device 1306 may be configuredfor storing the at least one first virtual object model.

Further, in some exemplary embodiment, the communication device 1302 maybe configured for receiving at least one second sensor datacorresponding to at least one second sensor 1316 associated with asecond vehicle 1314. Further, the at least one second sensor 1316 may becommunicatively coupled to a second transmitter 1318 configured fortransmitting the at least one second sensor data over a secondcommunication channel. Further, the communication device 1302 may beconfigured for receiving at least one first sensor data corresponding toat least one first sensor 1310 associated with a first vehicle 1308.Further, the at least one first sensor 1310 may include a first locationsensor configured to detect a first location associated with the firstvehicle 1308. Further, the at least one first sensor 1310 may becommunicatively coupled to a first transmitter 1312 configured fortransmitting the at least one first sensor data over a firstcommunication channel. Further, in some embodiments, the at least onefirst sensor 1310 may include a first user sensor configured for sensinga first user variable associated with a first user of the first vehicle1308. Further, the first user variable may include a first user locationand a first user orientation. Further, the communication device 1302configured for transmitting at least one first presentation data to atleast one first presentation device (not shown) associated with thefirst vehicle 1308. Further, the at least one first presentation datamay include at least one first virtual object model corresponding to atleast one first virtual object. Further, in some embodiments, the atleast one first virtual object may include one or more of a navigationalmarker (such as a navigational marker 1308, and/or a signboard 1604 asshown in FIG. 16 ) and an air-corridor (such as a skyway 1306 as shownin FIG. 13 ). Further, the at least one first presentation device mayinclude a first receiver configured for receiving the at least one firstpresentation data over the first communication channel. Further, the atleast one first presentation device may be configured for presenting theat least one first presentation data. Further, in some embodiments, theat least one first presentation device may include a first head mountdisplay. Further, the first head mount display may include a first userlocation sensor of the at least one first sensor 1310 configured forsensing the first user location and a first user orientation sensor ofthe at least one first sensor 1310 configured for sensing the first userorientation. Further, the first head mount display may include a firstsee-through display device. Further, the processing device 1304 may beconfigured for generating the at least one first presentation data basedon the at least one second sensor data and the at least one first sensordata. Further, the generating of the at least one first virtual objectmodel may be independent of the at least one second sensor data.Further, in some embodiments, the processing device 1304 may beconfigured for determining a first airspace class (with reference toFIG. 15 ) associated with the first vehicle 1308 based on the firstlocation including a first altitude associated with the first vehicle1308. Further, the generating of the at least one first virtual objectmodel may be based on the first airspace class. Further, in someembodiments, the storage device 1306 may be configured for storing theat least one first presentation data. Further, in some embodiments, thestorage device 1306 may be configured for retrieving the at least onefirst virtual object model based on the first location associated withthe first vehicle 1308. Further, in some embodiments, the storage device1306 may be configured for storing a second three-dimensional modelcorresponding to the second vehicle 1314. Further, the generating of thefirst presentation data may be based on the second three-dimensionalmodel. Further, in some embodiments, the communication device 1302 maybe configured for receiving an administrator command from anadministrator device. Further, the generating of the at least one firstvirtual object model may be based on the administrator command. Further,in some embodiments, the communication device 1302 may be configured fortransmitting at least one second presentation data to at least onesecond presentation device (such as the second presentation device 1320)associated with the second vehicle 1314. Further, the at least onesecond presentation device may include a second receiver (such as thesecond receiver 1322) configured for receiving the at least one secondpresentation data over the second communication channel. Further, the atleast one second presentation device may be configured for presentingthe at least one second presentation data. Further, in some embodiments,the processing device 1304 may be configured for generating the at leastone second presentation data based on the at least one first sensordata. Further, in some embodiments, the storage device 1306 may beconfigured for storing the at least one second presentation data.Further, in some embodiments, the storage device 1306 may be configuredfor storing a first three-dimensional model corresponding to the firstvehicle 1308. Further, the generating of the second presentation datamay be based on the first three-dimensional model. Further, in someembodiments, the at least one second presentation data may include atleast one second virtual object model corresponding to at least onesecond virtual object. Further, the generating of the at least onesecond virtual object model may be independent of the at least one firstsensor data. Further, the storage device 1306 may be configured forstoring the at least one second virtual object model.

FIG. 14 shows the corrected augmented reality view 1400. Further, theaugmented reality view 1400 may include a road drawn in the sky (such asthe skyway 1406) indicating a path that a civilian aircraft 1404 maytake in order to land at an airport. Further, the augmented reality view1400 may include the navigation marker 1408 indicating to a pilot 1402that the civilian aircraft 1404 should take a left turn. The navigationmarker 1408 may assist the pilot 1402 in navigating towards a landingstrip to land the civilian aircraft 1404.

Therefore, the corrected augmented reality view 1400 may provide pilotswith a similar view as seen by public transport drivers (e.g. taxi orbus) on the ground. The pilots (such as the pilot 1402) may see roads(such as the skyway 1406) that the pilot 1402 need to drive on. Further,the pilot 1402, in an instance, may see signs just like a taxi driverwho may just look out of a window and see road signs.

Further, the corrected augmented reality view 1400 may include (but notlimited to) one or more of skyways (such the skyway 1406), navigationmarkers (such as the navigation marker 1408), virtual tunnels, weatherinformation, an air corridor, speed, signboards for precautions,airspace class, one or more parameters shown on a conventionalhorizontal situation indicator (HSI) etc. The skyways may indicate apath that an aircraft (such as the civilian aircraft 1404) should take.The skyways may appear similar to roads on the ground. The navigationmarkers may be similar to regulatory road signs used on the roads on theground. Further, the navigation markers may instruct pilots (such as thepilot 1402) on what they must or should do (or not do) under a given setof circumstances. Further, the navigation markers may be used toreinforce air-traffic laws, regulations or requirements which applyeither at all times or at specified times or places upon a flight path.For example, the navigation markers may include one or more of a leftcurve ahead sign, a right curve ahead sign, a keep left sign, and a keepto right sign. Further, the virtual tunnels may appear similar totunnels on roads on the ground. The pilot 1402 may be required to flythe aircraft through the virtual tunnel. Further, the weatherinformation may include real-time weather data that affects flyingconditions. For example, the weather information may include informationrelated to one or more of wind speed, gust, and direction; variable winddirection; visibility, and variable visibility; temperature;precipitation; and cloud cover. Further, the air corridor may indicatean air route along which the aircraft is allowed to fly, especially whenthe aircraft is over a foreign country. Further, the corrected augmentedreality view 1400 may include speed information. The speed informationmay include one or more of a current speed, a ground speed, and arecommended speed. The signboards for precautions may be related towarnings shown to the pilot 1402. The one or more parameters shown on aconventional horizontal situation indicator (HSI) include NAV warningflag, lubber line, compass warning flag, course select pointer, TO/FROMindicator, glideslope deviation scale, heading select knob, compasscard, course deviation scale, course select knob, course deviation bar(CDI), symbolic aircraft, dual glideslope pointers, and heading selectbug.

Further, in some embodiments, information such as altitude, attitude,airspeed, the rate of climb, heading, autopilot and auto-throttleengagement status, flight director modes and approach status etc. thatmay be displayed on a conventional primary flight display may also bedisplayed in the corrected augmented reality view 1400.

Further, in some embodiments, the corrected augmented reality view 1400may include a one or more of other vehicles (such as another airplane1410). Further, the one or more other vehicles, in an instance, mayinclude one or more live vehicles (such as representing real pilotsflying real aircraft), one or more virtual vehicles (such asrepresenting real people on the ground, flying virtual aircraft), andone or more constructed vehicles (such as representing aircraftgenerated and controlled using computer graphics and processingsystems).

Further, the corrected augmented reality view 1400 may include anairspace. FIG. 15 is a chart related to the United States airspacesystem's classification scheme. Specifically, FIG. 15 illustratesvarious parameters related to one or more classes defined in the UnitedStates airspace system's classification scheme. The classificationscheme is intended to maximize pilot flexibility within acceptablelevels of risk appropriate to the type of operation and traffic densitywithin that class of airspace—in particular, to provide separation andactive control in areas of dense or high-speed flight operations. TheAlbert Roper (1919-10-13 The Paris Convention) implementation ofInternational Civil Aviation Organization (ICAO) airspace classesdefines classes A through G (with the exception of class F which is notused in the United States).

For an instance, a computing device (such as the computing device 1600)may analyze one or more parameters such as altitude, Visual Flight Rules(VFR), Instrument Flight Rules (IFR), VFR cloud clearance, and VFRminimum visibility etc. to determine an applicable airspace class.Further, the determined airspace class may be displayed on the virtualreality display. Further, the applicable airspace class may bedetermined using a location tracker such as a GPS and may be displayedas a notification on the virtual reality display.

Further, a special use airspace class may be determined. The special useairspace class may include alert areas, warning areas, restricted areas,prohibited airspace, military operation area, national security area,controlled firing areas etc. For an instance, if an aircraft (such asthe civilian aircraft 1404) enters a prohibited area by mistake, then anotification may be displayed in the corrected augmented reality view1400. Accordingly, the pilot 1402 may reroute the aircraft towards apermitted airspace.

Further, the corrected augmented reality view 1400 may include one ormore live aircraft (representing real pilots flying real aircraft), oneor more virtual aircraft (representing real people on the ground, flyingvirtual aircraft) and one or more constructed aircraft (representingaircraft generated and controlled using computer graphics and processingsystems). Further, the corrected augmented reality view 1400 shown to apilot (such as the pilot 1402) in a first aircraft (such as the civilianaircraft 1404) may be modified based on sensor data received fromanother aircraft (such as another airplane 1410). The sensor data mayinclude data received from one or more internal sensors to track andlocalize the pilot's head within the cockpit of the aircraft. Further,the sensor data may include data received from one or more externalsensors to track the position and orientation of the aircraft. Further,the data received from the one or more internal sensors and the one ormore external sensors may be combined to provide a highly usableaugmented reality solution in a fast-moving environment.

FIG. 16 shows an augmented reality view 1600 shown to a real pilot whilea civilian aircraft 1602 is taxiing at an airport, in accordance with anexemplary embodiment. The augmented reality view 1600 may include one ormore navigational markers (such as the navigation marker 1408) andsignboards (such as a signboard 1604) that assist a pilot to taxi thecivilian aircraft 1602 at the airport. The navigational markers mayindicate the direction of movement. The signboards may indicate thespeed limits.

The augmented reality view 1600 may help the pilot to taxi the civilianaircraft 1602 towards a parking location after landing. Further,augmented reality view 1600 may help the pilot to taxi the civilianaircraft 1602 towards a runway for taking-off. Therefore, a ground crewmay no longer be required to instruct the pilot while taxiing thecivilian aircraft 1602 at the airport.

Further, the augmented reality view 1600 may include one or more liveaircraft (such as a live aircraft 1606) at the airport (representingreal pilots in real aircraft), one or more virtual aircraft at theairport (representing real people on the ground, controlling a virtualaircraft) and one or more constructed aircraft at the airport(representing aircraft generated and controlled using computer graphicsand processing systems). Further, the augmented reality view 1600 shownto a pilot in a first aircraft may be modified based on sensor datareceived from another aircraft. The sensor data may include datareceived from one or more internal sensors to track and localize thepilot's head within the cockpit of the aircraft. Further, the sensordata may include data received from one or more external sensors totrack the position and orientation of the aircraft. Further, the datareceived from the one or more internal sensors and the one or moreexternal sensors may be combined to provide a highly usable augmentedreality solution in a fast-moving environment.

In accordance with exemplary and non-limiting embodiments, the processof acquiring sensor information from one or more vehicles, maintaining arepository of data describing various real and virtual platforms andenvironments, and generating presentation data may be distributed amongvarious platforms and among a plurality of processors.

In one embodiment, with reference to FIG. 21 , a centralized server 2100may serve as a repository for sensor information in a spoke-hub datadistribution model. As described elsewhere herein, a server 2100 mayreceive and store data from various vehicles 2104′, 2104″ indicative ofthe state of the vehicle. Examples of such data include, but are notlimited to, velocity, altitude, orientation, etc. The server may alsoreceive and store data from virtual vehicles 2106. In some instances,these virtual vehicles 2106 may operate with some human intervention.For example, an individual may operate a ground based flight simulatorwherein the simulated experience may be mapped to a virtual airspaceforming a database on the server. While the orientation, velocity,location, and the like are generated as attributes of a virtual entity,such data may be included in the sensor information repository andintegrated with the data describing physical vehicles and entities.

As described elsewhere, the server 2100 may likewise store informationdescribing one or more virtual objects. As with other objects/entities,these virtual objects may encompass a variety of attributes including,but not limited to, location, velocity, orientation, and various rulesdescribing the behavior and appearance of the virtual objects.

In some embodiments, each physical object, such as a plane, may bedescribed in the repository in both real terms and relative terms. Insome instances, relative terms may take the form of an offset value inthree dimensional space. For example, a first plane may be headingdirectly north over California at an altitude of 15,000 feet and a speedof 500 mph. At the same time, a second plane may be heading directlysouth over Germany at an altitude of 17,000 feet at a speed of 500 mph.In this example, it is desired that the two planes be enabled to engagein an air training exercise in which the two pilots fly in formation ina virtual airspace with the second plane approximately 50 feet off ofthe right wingtip of the first plane with both planes flying side byside at an altitude of 16,000 feet and headed due east over Japan.

In some embodiments, in accordance with this example, the server 2100may receive updated position and orientation data from each of theplanes indicative of the absolute position of each plane. For example,GPS coordinates of the first plane will be indicative of a location overCalifornia while GPS coordinates of the second plane will be indicativeof a location over Germany. The server may likewise maintain a databaseof a virtual airspace over Japan wherein each of the planes' actuallocations are translated into the coordinates of the virtual airspace.As a result, for example, the received latitude and longitudecoordinates of the first plane as it proceeds due north may betranslated into virtual coordinates over Japan whereby the first plane'sactual movement to the north is translated into movement due east.Likewise, movement by the first plane to gain altitude or lose altitudefrom its present actual altitude of 15,000 feet will be translated intodeviations about a virtual altitude of 16,000 feet. In a similar manner,the actual data received by the second plane may be stored as well astranslated into the virtual environment.

As a result, while each plane is thousands of miles from the otherplane, the server 2100 may send presentation information to each planeenabling the rendering of the other plane, such as via a pilot'saugmented reality display, as existing in a shared virtual environment.In addition to rendering various objects, terrain may be projected as ARcontent to one or more pilots such that one or more pilots operating ina shared virtual environment will experience the same virtualenvironment as existing above and about the same terrain. As describedmore fully below, the geographic extent of a virtual airspace will oftentimes be of an extent that is less than the clear airspace surroundingeach participating plane. For example, consider a disk-shaped virtualairspace that extends latitudinally and longitudinally in all directionsfrom a virtual center point for 50 miles at an altitude of 16,000 feetand extending to higher and lower elevations plus-or-minus 15,000 feet.When actual plane positions are translated into the virtual space, it ispreferable that the extent of the boundaries of the virtual space withrelation to each plane correspond to clear airspace around the physicalplanes in actual space. For example, as described, a first plane with anactual altitude of 15,000 feet may have a translated altitude in avirtual airspace of 16,000 feet. If the first plane is over the oceanoff the coast of California, any descent beyond 15,000 feet will placethe plane below sea level and may result in a potentially catastrophicsystem failure. However, even after descending 15,000 actual feet, thefirst plane exists in the virtual airspace at an altitude of 1,000 feet.

It is therefore preferable to map each actual vehicle to the virtualspace in such a way that the physical vehicle may move freely about thevirtual airspace without encountering any real world obstacles. Notethat in such instances, while each pilot in either the first or secondplane in this example may see a rendering of the other plane as avirtual image in, for example, an augmented reality display, the virtualairspace may appear quite different to each pilot. For example, thefirst pilot may see the second pilot off of his wingtip with the Sierramountain range beyond while the second pilot sees the first plane off ofhis left wingtip the lowlands of Bavaria in the distance.

In some embodiments, the virtual space may be defined to be smaller thanthe actual unobstructed, or “safe,” airspace of any of the vehiclessending sensor data to the server. Doing so may serve to avoid thepredicament of a pilot flying outside of the virtual airspace and beingimmediately confronted with a real world obstacle. In some instances,the amount by which a vehicle's safe airspace exceeds the dimensions ofthe virtual airspace may depend, at least in part, on a characteristicof the vehicle. For example, a vehicle capable of supersonic flight mayhave a greater excess and appended safe space as compared to a slowervehicle. In other instances, considerations such as the presence ofnational borders and/or restricted airspace may be taken into accountwhen establishing a suitable real airspace corresponding to a virtualairspace.

As described, data is being collected by sensors on vehicles 2104′,2104″ and transmitted to a central server 2100. This data is used todefine the state of all vehicles and objects, whether real or virtual,and to transmit presentation data to each vehicle to enable thepresentation of objects in a virtual manner. In some embodiments, thepresentation data may be provided to, for example, a gunner either inthe aircraft or in a ground vehicle via, for example, AR head gear. Insome exemplary embodiments, the processing of the data is distributedamong the processing platforms. Generating imagery for presentation to apilot may require the retrieval from memory of a wireframe model of anobject and surface textures to be draped upon the model. Depending onthe detail of the wireframe model and textures, it may be necessary totransmit several megabytes of data to a graphics card to create eachframe. In some instances, vehicles supporting processors with requisitegraphics capabilities may create the imagery for display to a pilotbased, at least on part, on data transmitted from the server to thevehicle.

For example, with reference to the preceding example, the server may mapthe location of a second vehicle to a place in the virtual airspacewhich is 50 feet off of the right wing of the first plane. The servermay transmit data in the form of a data structure to a processor on thefirst plane. Such data may include, at least, the position andorientation of the second plane. The data may represent the location ofthe second plane in relation to the first plane in absolute geographiccoordinates, as coordinates within a virtual airspace wherein each planehas information necessary to translate virtual airspace coordinates intoabsolute or relative positions in real space, or some mixture of thetwo. Retrieving onboard information detailing the position andorientation of the first vehicle as well as the view vector of thepilot's gaze, a processor on the first plane may inform the graphicsprocessing unit to create imagery for display to the pilot showing thesecond plane in a position and orientation received from the server.Likewise, the second plane may receive information transmitted from theserver detailing the position and orientation of the first plane and mayproceed to produce imagery for display to a second pilot showing thefirst plane in its proper relationship to the second plane.

As illustrated, the server 2100 functions as a central repository fordata defining the virtual airspace. If a virtual object or additionalvehicle is added to the database on the server 2100 representing thevirtual airspace, that object is effectively pushed out to all vehiclesfor display. In some embodiments, renderings for display created by aprocessor running on a vehicle may make use of graphic data storedlocally, stored on the server or some combination of the two. Forexample, the first plane may have stored locally a generic model andsurface textures for a generic F-22 fighter. At some point during anexercise, the server may push out portions of surface textures unique toa particular plane, such as a texture showing the name of the pilot asis commonly presented beneath the cockpit. Further, a wire frame modelof the actual pilot may be uploaded as well as stored locally byparticipating vehicles. As a result, in addition to transmittinglocation and orientation information for each vehicle to be displayed,the server may additionally send unique identifying information for aplane to be displayed. In response, when creating an image of the uniqueplane for display, each plane's processor may combine static model datawith data unique to each displayed vehicle to produce a more lifelikerepresentation.

In a similar manner, the server may continually push out updated displayinformation. For example, if a first plane manages to inflict a numberof virtual bullet holes in the fuselage of a dogfighting plane, theserver may push out to participating vehicles an updated portion of thesurface texture of the fuselage showing the bullet holes. In thismanner, data latency is reduced by reducing the amount of data that theserver 2100 needs to send to each vehicle 2104, 2106. By distributingthe processing, the server 2100 functions to coordinate the receipt andtransmission of data indicative of the state of the virtual airspace toeach interested entity and/or vehicle while the graphics functionsrequiring the movement of large volumes of data are performedefficiently by a plane's processor.

Because planes may be flying with respect to one another at speedsexceeding the speed of sound, vehicles, and objects, whether virtual orreal, may travel a perceivable distance between frames. For example, twoplanes closing on each other each traveling 500 mph (806 kph) results ina closing speed of 448 m/sec. If one is computing 50 frames per second,each plane will appear to have moved almost 9 meters with every newframe. As is evident, if the position and orientation data received byeach plane is delayed for even the briefest of time periods, thedisplayed position of a vehicle or object may be incorrectly plotted ormay appear to jump around rather than appear to be moving smoothlythrough space.

In some embodiments, historic and real-time data may be utilized asinputs to a performance model which may output extrapolated locationdata for objects.

FIG. 22 shows a method for utilizing historic and real-time data may asinputs to a performance model which may output extrapolated locationdata for objects, in accordance with some embodiments.

At 2200, the server may receive data indicative of a vehicle's pastposition in space. At 2202, the server may predict the vehicle'sposition into the future. For example, the server may fit a curvethrough a vehicle's discreet positions in space extending back in time,for example, for a number of seconds. Based on received sensor outputsfrom the vehicle and the historical data, the server may apply a modelto predict the position and orientation of the vehicle forward in timeat discrete points, for example, several seconds into the future. Insome embodiments, along with transmitting data indicative of theposition and orientation of various objects to each vehicle for display,the server may also send a plurality of future times and associatedpredicted positions and orientations for various objects as shown at2204.

For example, a first plane may receive time stamped position andorientation information at which to display a second plane. If thelatency between the current time and the time stamp is low, for example,1/1000 of a second, the first plane may create and display imagery fordisplay to the pilot of the first plane. The first plane may at the sametime receive a steady or intermittent stream comprised of a plurality ofextrapolated positions and orientations of the second plane. If, forsome reason, the most recently received actual position data for thesecond plane exhibits high latency (e.g., on the order of a second), orif an incoming data stream is compromised or broken, the first plane mayutilize previously extrapolated position data until data acquisition isrestored. In such an instance, it is possible that utilizing newlyacquired position data received while utilizing extrapolated data mayresult in the apparent position of a displayed object “jumping” from thelast extrapolated position to the newly identified actual position. Insuch instances, the system may operate where practicable to interpolatebetween the last utilized extrapolated position of an object and themost recent actual position of the object to provide for the appearanceof smooth movement of the object without any jumping.

In some embodiments, there may be provided a user activated kill switchto turn off the display of virtual objects. In environments wheremultiple photo realistic objects are displayed to a pilot, it may bepreferable to provide a method whereby the pilot only sees objects whichare physically occupying the same airspace. In some embodiments, thepilot or operator of a vehicle may enable an enhanced mode whereinobjects which are virtual and do not occupy the same airspace as thepilot may be visually tagged as virtual. For example, there may be threeplanes flying in formation in a virtual airspace. Two of the planes maybe in actual proximity to one another while the third may be flyinghundreds of miles away. Both of the proximate planes may see the thirdplane generated and displayed as a photo realistic object flying information in the virtual airspace. Likewise, the third plane may seeboth of the two proximate planes generated and displayed as a photorealistic objects flying in formation in the virtual airspace. Byactivating an enhanced mode, both pilots of the two proximate planes maysee the third plane rendered with a visual indicia indicating that it isvirtual. For example, the third plane may glow red, may be outlined ingreen, etc.

Operating in enhanced mode may allow each pilot individually todeclutter the observable airspace in order to focus on real worldobjects and obstacles.

In accordance with other exemplary and non-limiting embodiments, thedatabase maintaining the state of the virtual airspace may be accessedin real time and mapped to a physical location, such as an office space,for observation and interaction by one or more observers as illustratedwith reference to an exemplary and non-limiting embodiment at FIG. 23 .In one embodiment, visual indicia 2300 comprising markings may beapplied to a volume of space, such as a conference room or office, atknown locations enabling augmented reality display systems to integratethe display of virtual objects into the three dimensional volume ofspace.

For example, a virtual airspace 2304 comprising a cube ten miles on eachside may be mapped to second virtual display space 2302 comprising acube ten feet on each side wherein the virtual cube is further mapped toa physical volume of space in an office. As a result, planes flying inthe virtual airspace may be projected and displayed as occupying ascaled down version of the airspace within a ten foot by ten foot by tenfoot volume of the office. All objects stored as forming parts of thevirtual airspace may be represented in the virtual display space.Observers 2306 with augmented reality display systems 2308 may be ableto walk around the virtual display space 2302 and view the virtualdisplay space 2302 from different angles.

In some embodiments, observers may be enabled to interact with displayedvirtual objects and request more data. For example, an observer mayreach out and touch a displayed virtual plane causing a menu to bedisplayed in space allowing the observer to see information on the pilotof the displayed virtual plane. In other embodiments, a user may rewindto a previous moment in the display of the virtual airspace in order toview again a sequence of events.

In some embodiments, walking around the virtual display may occur duringa static moment of visualization such as, for example, during a freezeframe multi-domain exercise. In such instances, viewers may walk aroundthe displayed data in order to shift a point of view. In otherembodiments, viewers may employ a perspective shifting device, such as avirtual stick 2310. For example, a viewer may utilize virtual stickcontrols to manipulate a camera angle, a focal length and a positionallowing the viewer to fly anywhere and zoom in and out. In someembodiments, the viewer may shortcut these moves to “snap” into a POV ofany aircraft pilot. The viewer may shift time by using virtual stickcontrols such as rewind/fast forward, start/stop, repeat loops, reverse,slow motion and the like. In some embodiments, the viewer may “grab”objects in the scenario and “move” them temporarily to changepositions/orientations of aircraft while the scenario is playing back.

In other embodiments, the system may enable playback of recorded datafrom a repository of timestamped data describing various real andvirtual platforms and environments as they interacted in variousscenarios over a time period. For example, observers may project orotherwise view data from an aerial exercise comprising both real andvirtual entities as seen from the perspective of a pilot taking part inthe exercise. In some instances, the view point of the observer may beset to a point within the cockpit allowing for the observation of themotions of the pilot. As a result, it may be possible to observe, forexample, head and eye motions of the pilot. In exemplary scenarios wherethe recorded data include head, eye and plane attitude data tracked inreal-time, this allows for viewing the pilot's reactions during atraining exercise.

In addition to passively receiving virtual airspace data from theserver, observers may be enabled to interact with the system in order toalter the virtual airspace. For example, an observer may touch orotherwise indicate a portion of the virtual display airspace andindicate to the system the addition of three additional enemy fighteraircraft. These aircraft, once entered into the virtual airspacedatabase, will be pushed out to participating vehicles and entities asdescribed above.

Advanced Tactical Airborne Reconnaissance Systems (ATARS) may beutilized to provide real-time visualization of datasets to pilotstraveling at high speeds.

In some embodiments, the playback of recorded data may incorporate thedisplay of terrain. Such terrain may be displayed to provide context forthe positioning, motion and actions of a vehicle in a virtual or realairspace. In instances where the participating vehicles exist in thesame physical airspace, there may be displayed the actual terrain of theairspace. In instances where the participating vehicles exist in thesame physical airspace and the system operates to provide a virtualterrain via augmented reality, the virtual terrain may be presented toobservers of the playback.

As described above, two airplanes may be flying remote one from theother. For example, one airplane may be flying over the Pacific Oceanand one airplane may be flying over the Atlantic Ocean. Augmentedreality content comprising a virtual terrain of the mountains ofAfghanistan may be displayed to each pilot along with a virtualrendering of each alternate pilot to give the illusion that each pilotis flying in formation with the other pilot over Afghanistan. Whenobserving a playback of such an exercise, an observer may select theprojection of the virtual Afghanistan terrain common to both pilots asthe perceived terrain or may select a representation of either actualterrain over which one or both of the pilots flew.

In some instances, the displayed terrain may be enhanced for theobserver. In the above example, the system may have operated to displaya realistic rendering of the terrain of Afghanistan to each pilot.During playback, the projected terrain may be augmented with additionalgeospatial data to aid the observer. For example, the projectedAfghanistan terrain may by annotated with the position of anti-aircraftguns, troops and the like.

As described above, the present system operates to precisely identify aposition in space of a vehicle to enable the precise projection ofvirtual objects to an operator of the vehicle. In order to do so, it issometimes necessary to not only precisely define the location of aspecific point in the vehicle but also the small translations in spaceapplied to such a point to precisely locate the position and orientationof, for example, a pilots helmet. As a result, the system utilizes thederivation of the absolute position of the vehicle in space as well asrelative differences in position with respect to the absolute positionexhibited by, for example, a pilot's eyes. Utilization of this relativeposition information enables the system to project augmented realitydata to a pilot from the precise vantage point of the pilot's eyes.

In some exemplary embodiments, this method may be extended to provideprojected augmented reality data to more than one occupant of thesystem. For example, a GPS monitor, an accelerometer and an inertialguidance system may all be employed and their outputs combined toprecisely locate a point in the cockpit of an airplane. Further supposethat a tail gunner operating in the rear of the airplane is located, onevery model of the aircraft, precisely thirty feet behind the cockpitpoint. Utilizing this knowledge, the system may operate to provideaugmented reality data for presentation to a person occupying the tailgunner seat. In some embodiments, visual indicia may be placed inprecisely known locations in the aircraft and may be used to preciselyidentify a location and orientation of an occupant's eyes or viewingdevice. For example, three Xs may be placed about a tail gunner'sposition. The location of each X relative to a known position, such asthe point in the cockpit with a precisely derived absolute locationvalue, is known. The system may observe the location of the Xs, such asby a camera located on augmented reality goggles of the tail gunner, inorder to derive the location and orientation of the tail gunner'sviewing device. In this way, the ability to quickly and accuratelyderive the absolute location and orientation of a point in an aircraftmay be extended to similarly derive the relative location andorientation of various places within and about the aircraft. Thesederived relative locations may then be used to provide points of viewfrom which to generate virtual content for viewing by an occupant of thevehicle.

It is known to identify visual indicia in an environment wherein theindicia have known locations and subsequently using these knownlocations to present data to an augmented reality vision system. Forexample, a camera attached to augmented reality glasses may identify thefour corners of a known building face and proceed to present a visualoverlay tied to the surface of the building to a viewer. In otherinstances, a system may identify objects and their locations in spaceand proceed to present floating text around the objects thus providingadditional information to a viewer.

In contrast, in accordance with various embodiments described above, thepresent system operated to precisely define the location of a vehicleand an occupant of the vehicle without visual reference to any objectexterior to the vehicle. Further, as described above, the present systemallows for the determination of the precise location of a plurality ofoccupants of a vehicle.

As a result, the present system enables the presentation of virtualobjects and information to a plurality of vehicle occupants utilizingthe determined location and orientation of the vehicle without referenceto any outside landmark. For example, any number of bus riders mayselect a theme for presentation and experience an augmented realitydisplay tailored to the chosen theme. For example, a bus rider throughNew York City may select a theme devoted to how the city appeared in1920. While enjoying an otherwise normal bus ride experience, a riderwearing augmented reality glasses may look out the bus window to view apresentation of the surrounding buildings and landmarks as they wouldhave appeared in 1920. In some embodiments, only the viewing areadirectly in front of the viewer or in the direction of the viewer's gazeis augmented. As a result, wherever the viewer's gaze is directedappears to be as seen in 1920. In other examples, a viewer may choose aJurassic theme and see the surrounding environment augmented bydinosaurs. In some embodiments, the data associated with each theme tobe presented may be received form an entity owning or operated thevehicle. For example, a bus company may provide such an augmentedreality service for a fee or as a service to paying customers.

In some embodiments, the interior of the vehicle may be painted orotherwise visually altered in a known manner in order to aid in theproduction of augmented reality content. For example, if the interior ofthe bus is painted a known color of green, the system may be operated tonot present any augmented reality data over an area of augmented realityglasses corresponding to the shade of green.

As described herein elsewhere, the technologies of this disclosureinclude those that may be used to locate a vehicle, predict where thevehicle will be at a point in the future, locate a head-worn device of aperson in the vehicle, identify the orientation of the helmet, detectthe person's eye direction, and lock virtual content in a geospatialposition without the need for a physical world located marker foralignment.

Other exemplary and non-limiting embodiments relate to the placement oftravel information, advertisements, general information, location-basedinformation, and the like. In some embodiments, a computer process isadapted to enable an operator (e.g., advertiser) to make placements ofvirtual content such that the virtual content is properly positionedgeospatially. Once geospatially positioned a person or persons in avehicle or walking may use the technologies to observe the virtualcontent. There may be a user interface that enables a content poster,such as an advertiser, to place content with respect to somethingphysical in the environment. The process may convert the placement intolongitude, latitude and altitude/elevation such that a person with a HMDwill see it.

For example, an advertiser may operate, as through an interface, toenter information indicative of a mode of displaying information. Forexample, the advertiser may select, via a VR user interface, a portionof a building on which to project or otherwise display andadvertisement. Data may be entered defining an orientation, sourcematerial, data format, preferred time of projection and the like. Forexample, an advertiser may choose to have a static poster in .pdf formatdisplayed above the elevators at the Empire State Building from 9:00am-11:00 am. Likewise, the advertiser may choose to display a 3Drotating instance of a product displayed above the information kiosk inGrand Central Station from 5:00 pm-8:00 μm.

With reference to FIG. 29 , there is illustrated an exemplary andnon-limiting embodiment of a user interface for entering contentplacement data. As illustrated, a series of icons 2900 allow the user toselect, for example, a source of content to be displayed, a time whenthe content is to be displayed, a place where the content is to bedisplayed, etc. As illustrated, the user has selected the building shownin site selection window 2904. The user may rotate, enlarge and,generally, navigate through the model or point cloud to identify adesired location to place content. Likewise, a selected is displayed formanipulation in content window 2906. Content placement window 2902provides tools for zooming in and placing or painting the selectedcontent.

In some embodiments, content placement window 2902 may visuallyhighlight the display of areas available for content placement. Were thesystem to rely entirely on latitude, longitude and elevation to placecontent, very small errors in determining the placement coordinatescould result in an advertisement be displayed inches behind, andtherefore occluded by, a wall. In some embodiments, once the contentcoordinates are determined, the system displays the content over, infront of or on top of any occluding surface or object within proximityto the content.

In some instances, the advertiser may be presented with data indicativeof likely pedestrian traffic volume in the selected area as an aid toselecting the time and placement of materials. In some instances a userinterface may enable a user, such as a prospective advertiser, to see arendering of how the displayed material will look when implemented bythe system.

In some embodiments, once the nature and position of the displaymaterials has been defined, the system will render or otherwise convertthe placement coordinates of the materials to precise latitude,longitude and elevation coordinates for use as described elsewhere inthis disclosure.

As described above, the system may operate to ascertain with a highdegree of precision the location and attitude of the headsets of aplurality of occupants of a vehicle via, for example, ascertaining apoint in the vehicle with a high degree of precision and computing therelative location of each passenger's headset from the ascertainedpoint. With reference to FIG. 27 , there is illustrated an exemplaryembodiment of a vehicle 2700 containing multiple passengers 2702′. Asillustrated, each passenger 2702 is enabled to view displayed content2704 wherein each displayed content item is located at a uniquelatitude, longitude and elevation. As described herein, the smoothdisplay of elements in the environment is enabled, at least in part, bythe ability of the system to extrapolate the position of the vehicleinto the future and, hence, the position of each passenger, based, atleast in part, on various sensors from which the location, direction andspeed of the vehicle may be ascertained.

While some embodiments of this present disclosure relate to location andorientation estimates of a person's head and eyes within a vehicle, thedisclosure is not limited for use in vehicles. For example, with thelocation and orientation of a user's head and/or eyes (e.g., through anAR, VR, XR headsets) known and a prediction of the user's gaze positionat a near-future time, virtual content may be placed based on longitude,latitude, and elevation for an accurate viewing position.

Thus, in similar fashion, an individual with a HMD may be functionallyequivalent to a vehicle for purposes of the system tracking a currentand future position of the individual. With reference to FIG. 28 , thereis illustrated an exemplary and non-limiting embodiment of a person 2800wearing an HMD 2802 capable of implementing functionality describedelsewhere herein with regards to a VR or AR augmented helmet. Asillustrated, the individual is able to see content 2804 defined by, andeffectively anchored to, a position defined by latitude, longitude andelevation.

Disney

With reference to FIG. 30 , there is illustrated an exemplary andnon-limiting embodiment of the online platform as described elsewhereand applied to a scenario in which a user of the platform is traversinga populated expanse, such as a park, a city scape, a theme park and thelike.

As described elsewhere data may be collected providing a preciseposition of an individual in x,y,z space, or, alternatively, latitude,longitude and elevation. This precise position may be combined withinformation indicative of an orientation of a device located at theprecise position in order to display visual data, such as virtualobjects, of an operator of the device.

As illustrated, a theme park 3000 is comprised of various staticobjects, e.g., buildings, lamp posts, and the like, as well as movingobjects such as, for example, park attendees, service personnel and thelike. It is increasingly the case that such areas are covered by videocameras 3002. Because the position and orientation of each camera isknown with great precision and certainty, cameras 3002 may be used todetermine or to refine position information derived for deviceoperators. This is particularly true for operators in areas ofoverlapping coverage by one or more cameras via triangulation. Usingvideo and still imagery from the cameras, the system may operate toidentify the identity of individuals using the system. In someinstances, facial recognition may be employed. In other instances, a barcode, QR code or other such indicia may be affixed to an attendee to aidis visual recognition of identity.

In accordance with exemplary and non-limiting embodiments, cameras 3002may be any device operating to enable the calculation of positioninformation for device operators. For example, each camera 3002 positionmay also serve as a position of a base station operating with mmWavesignals in, for example, a 5G or 6G paradigm. Use of mmWave signalsallows for determinations of the position of a target using bothtrilateration and triangulation. Specifically mmWave emitters/receiversenable a determination of both the distance to and the angle between themmWave transmitter and the target. The resulting position determinationsmay be accurate on the scale of millimeters.

In some instances, the accurate position information determined via theuse of mmWave data signals may in turn be used to more accurately directthe mmWave beams to provide a reliable link for high data-ratecommunication. Such a link may increase the data throughput to thetarget enabling the provision of more voluminous and detailed AR contentto a target. As described above, use of mmWave beam technology may beused in conjunction with any other exemplary embodiment describedherein. For example, mmWave beam technology may be used to accuratelydetermine the position of moving targets, such as NASCAR automobiles asdescribed herein. Likewise, mmWave position determination may be appliedto scenarios involving the real-time determination of the position andorientation of athletes engaged in athletic events.

In another exemplary embodiment, a grid 3004 with known properties maybe adhered to a surface of the park or projected onto it. In someembodiments, the grid is painted onto the surface with a material thatreflects IR light exhibiting certain and known characteristics. As aresult, when sunlight reflects off of the grid, the system can see thegrid clearly by limiting viewing, such as via filters, to the narrowrange of exhibited wavelengths. In embodiments, the IR altering gridmaterial may be otherwise invisible in the visible wavelengths andtherefore not viewable by park attendees.

In addition to determining the position and location of people withinthe park 3000, the system may likewise observe, map and determine clearspaces within the park 3000 devoid of people or other objects. Thisdynamic designation of clear areas may be centrally stored andaccessible by AR and VR display systems of park attendees. This data maybe used to position virtual objects in real time in the AR displays ofattendees. For example, a patron may have a virtual assistant 3006 inthe form of a theme park character that guides or otherwise accompaniesthe attendee through the park. The illusion of reality is shattered if areal person traversing an open space can walk through the spacevirtually occupied by the virtual assistant 3006. The system may operateto only project a virtual object, such as a virtual assistant, in aspace that is free of the presence of dynamically determined traffic. Inother embodiments, a virtual assistant may react to the determinedpedestrian traffic adding a level of reality.

In some embodiments, all forms of position determination disclosedherein including, but not limited to, GPS, visual triangulation,accelerometers and the like may be combined to refine positioninformation. Because the area of a park is finite and includes manyobservable landmarks, static information describing the precise locationof various objects may be combined with the aforementioned forms ofposition determination. For example, a multitude of images may be taken,encoded with the positions of objects in the images and stored forretrieval. When, for example, it is determined that a person using an ARdevice, whether head-mounted or carried like a smart phone, is in anapproximately known position, images may be sent to the display devicethat reference the area surrounding the person. The display device maythen capture an image of the surrounding environment and compare it toreceived and encoded images. By a process of matching what is seen inreal time from the AR device with the statically stored and encodedimages, the AR device may precisely determine its position withreference to encoded position information of nearby objects.

In contrast to the many aerial examples disclosed above, it is notnecessary that each viewer utilizing an AR display observe virtualobjects as appearing at the same place in space. For example, twodifferent people each observing a personalized digital assistant 3006may each see their assistant as occupying the same actual space. Asneither observer sees the other's assistant, there is no overlap.

In some embodiments, a predetermined set of observers may be linked withall linked observers seeing the same virtual objects. For example, afamily of five may all see the same assistant as it guides them allthrough the park. In some instances, a digital assistant may note when achild is far from the others in the group and may operate to encouragethe “lost” child to follow the assistant to another member of thepredefined group.

In accordance with other exemplary and non-limiting embodiments, AR andVR displays may be utilized on moving rides, such as roller coasters, ina manner similar to that disclosed herein with regards to ATARSimplementations. Specifically, the latitude, longitude and elevation ofa user's head may be precisely determined using any of the modalitiesdiscussed herein. Likewise, the precise location of a display device,such as a smartphone, may be determined. At the same time, theorientation and viewing directions of the display devices may bedetermined.

In contrast to determining the precise location and orientation ofaircraft traveling at potentially supersonic speeds within large volumesand following trajectories and paths which are determined in real timevia the control inputs of a pilot, many rides, such as roller coasters,follow a well defined path. Sensors implanted within the physicalhardware of the ride may provide position data to the displays.Likewise, visual cues and markers may be distributed throughout andabout a ride to provide for precise orientation and positionmeasurements. As discussed elsewhere, the system may use currentmeasurements of velocity and orientation to extrapolate into the futureto accurately predict the future position and orientation of a user'sdisplay device.

In some embodiments, knowledge of a generalized path may be utilized toaid in determining position. For example, some relatively slow movingrides, such as boat trips, follow a generally planned route with slightdeviations from side to side. These deviations, while somewhat random,occur within a constrained space that limits the magnitude of thedeviations. In some instances, sensors making use of, for example,visual cues may be used to determine position and orientation data. Forexample, visual examination of a boat as it passes by a point ofgenerally known location may be utilized to precisely determine theboat's position and orientation. Once known, visual cues within the boatmay be used to precisely determine a position and orientation of apatron's display device in much the same way as described above withreference to a tail gunner within a plane whose cockpit location andorientation has been precisely determined.

In some embodiments, AR related data may be displayed to a patron tomore efficiently move the patrons around the park. For example, thesystem may note that a group of individuals collectively areexperiencing via their AR displays a personalized digital assistant 3006in the form of a beloved cartoon character. It may also be noted that ashow is about to begin in ten minutes in an auditorium that is fiveminutes away from the group. As a result, the system may operate tocause the personalized digital assistant 3006 to suggest that theyattend the show and may interact to confirm acceptance. In someinstances, the system may use the precise positioning aspects describedherein to project a snippet of the show onto a nearby building or onto avirtual screen viewable by the group in order to generate excitement forthe show. In some instances, the AR displays of the group may displayvirtual markers, such as arrows or a bouncing ball, to direct them totheir destination.

IMU and LIDAR Combination

As described above, the head tracking system may identify the positionof a person's or persons' heads within a known environment. Prior artuse of conventional head tracking solutions is generally too slow, notaccurate enough, and/or error prone to mention a few. Working to locatea pilot's head within a cockpit of an airplane, the inventors discoveredthat electromagnetic noise in the cockpit is difficult to manage and maycause significant errors when using electromagnetic locationtechnologies. The inventors further discovered that using infrared lighttriangulation is also prone to errors due to the highly reflectivenature of the environment.

As a result, exemplary and non-limiting embodiments relate to a datafusion computer process using Lidar and inertial measurement unit (IMU)data feeds for the estimation of a head position within a knownenvironment as illustrated at FIG. 24 . An IMU 2402 may be mounted onthe helmet 2400 of a user (e.g., pilot) and the IMU may track themovements of the head such that the location of the head may bepredicted. Simultaneously, Lidar 2404 may track the user's head. The twodata feeds may be fused to accurately track the head position andmovement. The process involves tracking the head movements using the IMUand then correcting IMU drift by comparing the IMU predicted positionwith a Lidar determined position. The periodic calibration of the IMUprediction with the Lidar location is done throughout the trackingprocess leading to a very fast determination of the head position (e.g.,less than 5 ms).

Lidar generally uses non-visible light to measure time-of-flight timesto generate three dimensional maps of an area. In embodiments, the knownenvironment has been pre-mapped, and the Lidar is used to measure,through time-of-flight measurements, the distance between the person'shead and known positions within the known environment. The Lidaridentifies three or more areas for inclusion in a location assessment(e.g., for triangulation). However, Lidar measurements, even in theknown environment, are generally too slow to make a seamless contentpresentation in AR. Lidar generally refreshes its location calculationsabout 5 times per second.

IMU processing is very fast, but it drifts over very short periods oftime, so it is not a reliable location system for AR. However, IMU basedlocation predictions are done very fast, generally around 1000 time persecond. With the IMU data fused with the Lidar data, the locationmeasurements can be measured very close to the IMU rate itself, with acalibration with the Lidar data being completed based on the Lidarrefresh rate. In this manner, very accurate Lidar processing may be usedto precisely and periodically recalibrate a starting point to which IMUdeviations in position may be applied. The combination of rapid IMUupdates periodically corrected with Lidar data serves to continuallymitigate potentially unacceptable errors caused by IMU drift.

FIG. 19 is an exemplary and non-limiting embodiment of a fighter jetcockpit with computer generated jets 1906, 1908 and 1910 presented to apilot of the fighter jet through augmented reality. The pilot of thefighter jet is flying a real aircraft and is seeing computer generatedassets through a see-though computer display such that the pilot can seethe outside environment as well as the computer generated content, whichappears to the pilot to be in the outside environment. A Lidar systemmay be used to track certain features 1912 within the cockpit and maketime-of-flight measurements between the helmet, or other portion of thepilot or pilot's gear, and the features as a method for tracking theposition of the pilot's head within the cockpit. In embodiments, theLidar may be mounted in the cockpit or on the helmet or otherwise headmounted.

In embodiments, the Lidar may be mounted on the helmet or other headmounted system. The Lidar may then make its time-of-flight measurementsbetween the helmet and points detected within the cockpit. The cockpitmay be pre-mapped, so the Lidar does not need to re-map the area, but,rather, identifies known points within the pre-mapped cockpit to whichto measure. The Lidar may have a set of points within the pre-map thatit generally selects from to increase the speed of the Lidar process.For example, rather than the Lidar making time-of-flight measurements todifferent parts of the cockpit it may have a preferred set of pointswithin the cockpit that it looks for. In the event those pre-identifiedpoints are not detectable (e.g., because of interference) the Lidar maymake measurements to other parts of the cockpit. The arrangement of thepreferred points in the cockpit may be based on separation distancebetween the points themselves and/or the points and the helmet mountedLidar to increase the accuracy of the Lidar data as used fortriangulation or other calculation.

In embodiments, the Lidar may be mounted in the cockpit and positionedto track the helmet. The helmet, or other head mounted system, may haveidentifiable features that the Lidar can identify and track with itstime-of-flight measurements.

FIG. 20 illustrates a pilot's helmet 2000 with a time-of-flightmeasurement system (e.g., Lidar) 2002 and an inertial measurement unit(e.g., IMU) 2004. The Lidar 2002 may make periodic measurements withinthe pilot's environment to locate the helmet's position and identify inwhat direction the pilot is apparently looking as indicated by theposition of the helmet. The Lidar time of flight measurements arerelatively slow but they are highly accurate. The IMU 2004 may makemeasurements to track movements of the helmet, which can be mapped toidentify the helmet's location and position. The IMU measurements arerelatively fast and accurate, but they accumulate error by ‘drifting’.

The two data feeds, time-of-flight and inertial measurements, may bemerged for analysis or separately analyzed with reference between thetwo such that the IMU location and position calculation is compared tothe time-of-flight location and position calculation at a coincidentalor near coincidental time(s) of data acquisition. The comparison may beused to re-calibrate the IMU location and position calculation. There-calibration may be done each time the Lidar and the IMU have dataacquisitions at coincidental or near coincidental times.

The vehicle (e.g., fighter jet, bus, car, truck) may include an IMU tomonitor the vehicle's movements. In embodiments, the IMU data from thehead tracking system may be compared to the IMU data from the vehicle'smovement such that movements of the helmet, or other head mountedsystem, can be separately derived from the vehicle's movement. Ineffect, the movement of the vehicle as measured by a vehicle IMU may besubtracted from the movement of the IMU of the head tracking device toderive movement of the head device IMU relative to the vehicle and notto some external frame of reference beyond the vehicle. An augmentedreality system as described herein may need the separate IMU datacompared such that the location, attitude and force vectors of the planecan be used separately from the estimates of the head location, attitudeand force vectors. For example, the plane's IMU may be used tounderstand where the plane is within a mapped virtual environment andthe helmet's IMU may be used to understand where from within the planethe pilot is looking.

In some embodiments, a pre-map data set (e.g., point cloud data set) maybe referenced in the process of head tracking with Lidar. A pre-mapremoves the necessity of the Lidar to actively map the environment,which speeds up its distance measurement refresh rate. A head-wornsystem with Lidar may be matched or keyed to a type of vehicle orparticular vehicle. The systems may confirm the key (e.g., throughBluetooth) and then the Lidar system may operate based on theunderstanding that it has the correct map for the environment. The key,may involve a menu, where a user may select the vehicle to which it ispaired. The menu may have a listing of all accessible pre-mappedenvironments. In the event that a user is getting into an unknownvehicle, the user may be prompted to select a vehicle from a menu as theuser gets near or into the vehicle. For example, a public transportationbus, train, etc. or a commercial airliner, car, etc. may be connectableto the head mountable system. Once connected, a map to the otherwiseunknown vehicle may be made available to the Lidar system (e.g.,downloaded to the HMD, connectable via a wireless connection).

With reference to FIG. 17 , a system consistent with an embodiment ofthe disclosure may include a computing device or cloud service, such ascomputing device 1700. In a basic configuration, computing device 1700may include at least one processing unit 1702 and a system memory 1704.Depending on the configuration and type of computing device, systemmemory 1704 may include, but is not limited to, volatile (e.g.random-access memory (RAM)), non-volatile (e.g. read-only memory (ROM)),flash memory, or any combination. System memory 1704 may includeoperating system 1705, one or more programming modules 1706, and mayinclude a program data 1707. Operating system 1705, for example, may besuitable for controlling computing device 1700's operation. In oneembodiment, programming modules 1706 may include image-processingmodule, machine learning module and/or image classifying module.Furthermore, embodiments of the disclosure may be practiced inconjunction with a graphics library, other operating systems, or anyother application program and is not limited to any particularapplication or system. This basic configuration is illustrated in FIG.17 by those components within a dashed line 1708.

Computing device 1700 may have additional features or functionality. Forexample, computing device 1700 may also include additional data storagedevices (removable and/or non-removable) such as, for example, magneticdisks, optical disks, or tape. Such additional storage is illustrated inFIG. 17 by a removable storage 1709 and a non-removable storage 1710.Computer storage media may include volatile and nonvolatile, removableand non-removable media implemented in any method or technology forstorage of information, such as computer-readable instructions, datastructures, program modules, or other data. System memory 1704,removable storage 1709, and non-removable storage 1710 are all computerstorage media examples (i.e., memory storage.) Computer storage mediamay include, but is not limited to, RAM, ROM, electrically erasableread-only memory (EEPROM), flash memory or other memory technology,CD-ROM, digital versatile disks (DVD) or other optical storage, magneticcassettes, magnetic tape, magnetic disk storage or other magneticstorage devices, or any other medium which can be used to storeinformation and which can be accessed by computing device 1700. Any suchcomputer storage media may be part of device 1700. Computing device 1700may also have input device(s) 1712 such as a keyboard, a mouse, a pen, asound input device, a touch input device, a location sensor, a camera, abiometric sensor, etc. Output device(s) 1714 such as a display,speakers, a printer, etc. may also be included. The aforementioneddevices are examples and others may be used.

Computing device 1700 may also contain a communication connection 1716that may allow device 1700 to communicate with other computing devices1718, such as over a network in a distributed computing environment, forexample, an intranet or the Internet. Communication connection 1716 isone example of communication media. Communication media may typically beembodied by computer readable instructions, data structures, programmodules, or other data in a modulated data signal, such as a carrierwave or other transport mechanism, and includes any information deliverymedia. The term “modulated data signal” may describe a signal that hasone or more characteristics set or changed in such a manner as to encodeinformation in the signal. By way of example, and not limitation,communication media may include wired media such as a wired network ordirect-wired connection, and wireless media such as acoustic, radiofrequency (RF), infrared, and other wireless media. The term computerreadable media as used herein may include both storage media andcommunication media.

As stated above, a number of program modules and data files may bestored in system memory 1704, including operating system 1705. Whileexecuting on processing unit 1702, programming modules 1706 (e.g.,application 1720 such as a media player) may perform processesincluding, for example, one or more stages of methods, algorithms,systems, applications, servers, databases as described above. Theaforementioned process is an example, and processing unit 1702 mayperform other processes. Other programming modules that may be used inaccordance with embodiments of the present disclosure may include soundencoding/decoding applications, machine learning application, acousticclassifiers etc.

Asset operators, ground troops and others involved in military combatmay find themselves in complex situations and they may have to make aseries of decisions in quick succession to accomplish a mission. Theseindividuals may have a plan and a leader, but each one, or groups ofpeople, still have to make individual decisions based on their trainingand information that they have about the situation. Communication andadherence to validated tactics is vital in such situations andinsightful guidance provides a path to success. AI systems may processvast amounts of combat field data and provide insightful guidance toindividuals, groups, leaders, etc. while they are being trained andwhile they are in combat situations.

There are many combat situations where AI systems may provide usefulsuggestions to military personnel in training and combat situations. Forexample, in accordance with an exemplary and non-limiting embodiment, afighter pilot may be on a mission to escort and protect a strike packageon a mission. The flight may encounter enemy fighters approaching todisrupt the package's mission. The escorting fighter pilot(s) has tomake a decision on how to deal with the incoming fighters. The enemy maybe a simple configuration of a manageable few assets, but the enemy maybe a well-organized force with an advanced Integrated Air Defense System(TADS). The fighter pilot, and his flight, must manage this complexsituation to accomplish the mission and avoid losses.

The series of decisions and events leading up to and followingengagement of the enemy can be thought of as a series of decisions in atimeline. The pilot may have different information to consider dependingon his position relative to other assets, including his team members andthe enemy, at each point along the timeline.

The following is an example of how such engagement decisions may be madeand upon what type of information the pilot can use. First, the pilotreceives information through sensors, such as radar, indicating thatenemy combatants are incoming. At this point in the timeline, the pilotmay not be able to visually see the enemy because they are beyond visualrange (BVR). The pilot therefore relies on radar and other information.The radar and other information may be derived through sensors on theaircraft or remote systems (e.g. airborne early warning and control(AWACS), ground support, etc.). The pilot may also communicate withothers that have more information on the enemy. Given this information,criteria may be met that requires the flight to commit forces tointercept the enemy in order to protect the strike package.

The pilot may decide to fire an air-to-air missile targeting the enemywhile still BVR. Again, while BVR, prior to the launch of the missile,the pilot relies on sensor data. The pilot may then monitor sensors orlook for an explosion in the air to indicate success. If the missilemisses, the pilot has to make another decision. Does he shoot another?Does he continue on the path to close intercept? Does he wait for morehelp? etc.

Generally speaking, the pilot is looking to remove the enemy dangerwithout close engagement. Close engagement (e.g. within visual range(WVR)) becomes even more complicated. And it comes with more informationfor the pilot to consider including all of the visual information.However, the pilot may enter close combat and make more fast decisionsbased on all of the information at hand.

Once the enemy engagement is removed, the pilot may need to find his wayback to escort the strike package or move to intercept a new threat. Hemay find himself many miles from either. He then absorbs the informationhe has and makes the next series of decisions.

Described herein are systems and methods for training pilots in realaircraft in combat situations. The combat situations may be verycomplicated, as indicated in the example above, or they may be morestraightforward, such as learning to refuel in the air. The trainingsimulations may involve many friendly and enemy assets on the ground, inthe air, in space, etc. As described herein, an augmented reality systemprovides a synthetic environment for training WVR as well BVR. Theaugmented reality system may provide the pilot with a see-throughhead-mounted display, as described herein elsewhere, such that the pilotcan see through the display but also be presented with virtual content.The virtual content may be assets (e.g. other aircraft) within thesimulation. With the pilot experiencing a synthetic environment thatincludes simulated activities WVR and BVR, the pilot may train for thesecomplex situations.

Artificial intelligence, machine learning, deep learning, etc. (“AI”)may be used to help a pilot, or other operator, make decisions while insimulations or while in real combat situations. A training and combatinformation platform that provides a pilot with an environment, which isa combination of live assets (e.g., a real asset), virtual assets (e.g.,computer generated and controlled) and constructive assets (e.g.,computer generated and human controlled), that spans distances from wellbeyond his vision to being up close and personal. This environment maynot only be used for training a pilot, but it can be used to train AIsystems for improved training and combat information and guidance.

An AI system according to the principles of the present inventions maycontrol training simulations. The training simulation may be presentedto a pilot while the pilot is in a real aircraft flying in an airspace.The simulations may involve the presentation of data, communications,etc. to represent assets BVR of the pilot and WVR of the pilot. Thepilot may then run through many simulations where he maneuvers his planeto perform a mission while managing enemy and friendly assets. While thepilot is engaged in the simulations he may be monitored and recordedthrough sensor feedback, his plane's maneuvers may be tracked andrecorded, and the maneuvers of the other assets in the simulation may betracked and recorded. The recorded data from many simulations may beused to train the AI systems that control the virtual assets. The AIsystems may learn from the pilot's experiences, head position, eyeposition, bio-indicators from the pilot, the pilot's maneuvers, enemymaneuvers, friendly maneuvers, etc. to better predict what movementsmight be made and how to guide a pilot in similar situations. Thetrained AI systems can then be used to further train pilots and providepilots with real time suggestions in a combat situation or to help thepilot perform a mission.

The AI guidance and cues presented to the Pilot during training oractual missions may be audio, visual (e.g., AR), haptic, or other. Thepilot may receive audio guidance, information, cues, alerts, etc. basedon the AI systems understanding of a complex situation. The audio mayprovide the pilot information directly from the AI system, which iscomputer generated content. The audio may be coming from a human on theground or elsewhere where the human is processing recommendations fromthe AI system and/or consenting to AI suggested actions. The pilot mayalso or instead receive visual information that is presented on aheads-up display, head worn AR display, on an instrument panel, etc. Thevisual information may come directly from the AI system. It may includevisual cues indicating navigation guidance, maneuver guidance,restricted zones (e.g. country restricted no-fly zones, an occupiedairspace (e.g. occupied by another plane)), mission targets, incomingthreats, etc.

An AI system according to the principles of the present invention mayinclude multiple separate and coordinated systems using multiple AIsystems depending on the situation. As discussed herein elsewhere,assets WVR produce at least one very significant extra informationstream as compared with BVR; namely, visual information. The environmentalso significantly changes for the pilot once he is within visual rangeof an enemy, it becomes less predictable and the situation can changevery quickly. The AI system WVR is gaining an understanding of thesituation based on the additional information that the pilot sees,feels, hears, etc. The WVR AI system uses this additional perspectiveand information to give what may be different from what might beprovided in a BVR situation. So, there may be a WVR AI system and methodand there may be a BVR AI system and method. The two AI systems andmethods may need to coordinate because what is happening in oneenvironment may effect what is happening in the other.

In addition to coordinating an AI system WVR and AI system BVR, adifferent AI system may be invoked at a transition point between BVR andWVR. The transition AI may have different rules and processes thaneither the BVR or WVR due to the nature of the environment. With BVR thepilot generally relies on instrument feedback and guidance. With WVR anAI system in accordance with the present disclosures may use rules andprocesses inclusive of the nature of close combat. As an enemy assetapproaches WVR the pilot must get ready for the WVR experience.Preparation may include identifying where, within the pilot's visualfield, the enemy is going to approach from, how quickly the enemy isgoing to be approaching or passing, the attitude and direction of theenemy asset, what maneuver the enemy may make in transition or once WVR,etc. The pilot's senses may also be heightened in the transition periodbecause he is preparing for a close engagement. The transition AI maytake into account all of the preparation information and the pilot'sheightened senses when providing guidance to the pilot or plane.

Similarly, the AI control system of virtual assets (e.g., computergenerated and controlled) may also have different rules and processesfor the various distance-based scenarios. Such AI control may be basedon different conditions and anticipated conditions in BVR, WVR, and inthe transition range.

A pilot may be operating in a live aircraft and performing trainingsimulations. Virtual and constructive assets may be presented to thepilot during the training exercises. The virtual assets may becontrolled by an AI system with coordinated AI for BVR, WVR and thetransition between BVR and WVR. The virtual asset AI control may behavedifferently in each distance-based scenario. For example, as an enemyvirtual asset approaches the live asset, within the virtual environment,or another virtual or constructive asset, the enemy asset may operateunder AI processes that take into consideration that, if the enemy assethad an actual enemy pilot controlling the asset, the pilot would have tomake certain preparations and his senses would be heightened.Consideration may also be given to the anticipated increased cognitiveload on the pilot. This could provide a virtual asset control that moreclosely mimics a live asset with a real pilot during simulations.

The transitional AI controlling a virtual asset in a simulation mayunderstand that the virtual asset is a type that is to be consideredautonomous. In this situation, the transitional AI may control thevirtual asset based on preparing to go into WVR mode, but it may notconsider the pilot's cognitive load or heightened senses.

A simulated or real combat situation may involve many assets WVR and/orBVR of a pilot. There also may be more than one pilot being assisted byan AI system. Each pilot has its own WVR range and the respective WVRranges may overlap. The fact that one pilot may be WVR of an asset,causing that one pilot to process the additional visual information, mayneed to be considered by the AI system when providing information toanother pilot that may not have anyone, or a different asset, WVR.

With reference to FIG. 18 , there is illustrated an exemplary andnon-limiting embodiment of a situation with assets in various positions.As illustrated, three real, or live, aircraft are depicted as “R”.Virtual assets are represented as “V”. Constructive aircraft aredepicted as “C”. As can be seen, the three real aircraft may havedifferent assets their WVR, each asset has restricted maneuverability,and each asset outside of anyone R's WVR may need to be considered byits BVR AI controlling system. As a result, the WVR AI system may bedeployed for one asset and not another or two or more assets may beadvised by a WVR AI model and others may be advised by a BVR model. Eachmodel may affect the other as well.

AR for Obstructed Views in the Grandstands

NASCAR, F 1, and IndyCar are all very fast-moving sports with huge fanbases. Thousands of fans pack road track side grandstands to get aglimpse of their favorite driver speed past. It is thrilling to see thecars fly by while they are battling with their competition.Unfortunately, fans don't get to see the cars for too long as the tracksare very large by comparison to other sports such as football orbaseball. As a result, they only get to see a portion of the track. Anaugmented reality (which may be augmented reality, virtual reality,mixed reality, etc.) system may be used by fans in the grandstands tobetter ‘see’ the track and the cars.

As described herein, an AR system for fast moving vehicles may involve atracking and prediction system that precisely estimates the location,attitude, and other conditions of a vehicle and a driver's head positionin the same manner as described above with reference to pilots andplanes. Such a tracking and prediction system may be used to deliver afan-based AR experience. A fan may have an AR device (e.g., a phone,tablet, head mounted device with a see-through screen, head mounteddevice with a fully immersive screen) and may use it to ‘look’ atportions of the track that are otherwise obstructed or too far to seewell. If the device is a hand-held device, the fan may point the cameraof the device towards the section of the track that is of interest. Ifthe device is a head-mounted device, the fan may be able to simply lookin the direction of interest to see the other portions of the track.They may then “see” the other portions of the track through a digitalaugmentation of the environment. The digital augmentation may includedigital representations of the cars on the track. So, the fan may beable to simply look out to an obstructed view of the track and see acomputer-generated view of the track and the cars racing on the track.

Especially with live sports, it is important to have good alignmentbetween digital representations of the cars and track with their realpositions. Otherwise, the fan might see ‘jitter’ or misalignment betweenthe digital content and the real car when the real car is visible, suchas in a transition area. For example, the car may be a quarter mile awayand not visible to the fan. The fan may be looking at the ARrepresentation of the car and track. As the car reaches a transitionpoint where it is visible to the fan, the digital image should bealigned with the actual car to make a most enjoyable experience.

Latency is an enemy of good AR alignment with fast moving objects, asdiscussed elsewhere herein. As a reminder, with a very good predictionof where the car is going to be in a very short period of time in thefuture, say 100 ms, the AR content can be rendered based on the futureposition and time and presented at the predicted time for alignment ofthe content with the fast-moving car. A central computer system may betracking and predicting the near-future locations of each of the cars ina race such that the central system can communicate AR content to thefans in the stadium.

It may be important to maintain a good understanding of the location ofthe fan using the AR device and the car position and/or near future carposition. The alignment between the near-future position and the fan'sposition and head/eye viewing direction may determine the placementlocation of the AR content on the computer display of the fan's ARdevice. The AR device may have GPS, a compass, IMU, accelerometers, etc.to help locate the device and track its position. The AR device may alsohave an eye tracking system to estimate the direction of the fan's eyesfor more precise placement of the AR content in the device screen. Thefan's AR device may use inside-out, outside-in, or other tracking systemto assist in determining its location and direction. Inside-out andoutside-in tracking can be relatively slow, but it is capable ofproviding an acceptable experience because the fan is moving relativelyslowly. A fan device tracking system may use markers in the environment(e.g., on the seats, stadium structural components) such that the devicecan track its position in relation to the markers. Seat position itselfcould also be used to determine the fan's seat position. The fan mayconfirm that he is in the seat or an automated system (e.g., GPS,inside-out, outside-in) may estimate that the fan is likely in his seatand then the fan's ticketed seat number may be used to refine hisposition estimate by comparing the seat position to a map of thestadium.

The track layout itself may be pre-mapped based on actual geospatiallocations. This creates absolute references to the track. The absolutetrack references can be used in the generation of the AR content. Forexample, the system may calculate a near-future position of three carson the track. The near-future position of the car may then be associatedwith the track at the near-future position. This can create alignmentbetween the near-future position of the car and the track such that theuser experience aligns with reality. Without good track alignment, forexample, the car may look like it is turning into a corner while thetrack still has a straight appearance. This may be confusing to a fanthat understands the physics of the car.

The AR system may also be used to augment a fan's view of cars on atrack that are in view of the fan. Information such as speed, runningorder, engine conditions, tire conditions, pit information, etc. may bepresented with accurate content placement associated with the vehicle ofinterest. The augmented view may also include graphic depictions ofparts of the car. Brakes may be highlighted in red. The motor,suspension, drive chain, fuel load, etc. may be graphically highlighted.

Virtually Racing Against a Real Car

An embodiment of the present invention may include an AR/VR/XR videogame where a user can race against or with a professional driver duringan actual race or other event. This may be a fan experience in thegrandstands, or it may be a separate experience. Since the system knowswhere the car is, how it is positioned and where it is going to be inthe near-future, one may generate an avatar of the car and position iton a virtual track that represents the actual track at the near-futuretime. For example, the avatar may be a 3D model of the actual car,including performance specifications, appearance, etc. The user of thesystem may have computer user controls (e.g., simulated steering wheel,gas pedal, brake pedal, nitrous injection) and may be positioned to viewthe avatar from behind. The user could then follow behind the avatarduring a real race or other event. The system may be used in a “follow”mode where the user position is automatically controlled to follow theavatar. It may also be in a “race” mode where the user may use hiscontrols to try to maintain position behind the avatar or even overtakethe avatar. The game may include the presentation of several avatarsrepresenting several actual race cars in area.

In embodiments, the user's virtual car may bump or otherwise interactwith another virtual object (e.g., a curb, guardrail) or an avatar. Theinteraction may cause the user's virtual car to suffer a consequence(e.g., slowing, rolling, abruptly turning). For example, a user mayattempt to overtake an avatar and the user may virtually hit the avatar,which may cause the user to have to take his foot off the gas, slowingthe car so he can maintain control. Conversely, he may crash.

In embodiments, the user may overtake an avatar, possibly when the carrepresented by the avatar has a mishap, pits, or when the user is justso good he made a pass. The game may then allow the user to chase thenext car in line in front of him or select another driver to raceagainst.

In embodiments, there may be more than one user racing against one ormore real cars represented by avatars. The users may interact with eachother (e.g., bumping, hitting, crashing) while they chase the avatar(s).Each user may see the other users and the other avatars when they are ina virtual position with respect to one another that they would normallyhave a view in real life. A winning scenario may be whomever overtakesthe avatar or most avatars wins the race. Another winning scenario maybe the user with the closest finish to the avatar(s). Of course, otherwinning scenarios may be programmed and are envisioned by the inventor.

Sport Practice in AR using Volumetric Content Positioning

Practicing a position in a team sport tends to require the team to gettogether. There are times when individuals can practice on their own,but the experience is very different, and many things cannot be practicealone or with a limited number of team members. Practicing on a realfield using augmented reality can be used to simulate a team, limitedteam or individual practice sessions.

Existing ‘inside-out’ and ‘outside-in’ technologies are limited and, inmany situations, unusable for real-field simulated practice sessions.

GPS and Future Position Estimation

With reference to FIG. 25 , there is illustrated an exemplary andnon-limiting embodiment of an application of the disclosed technology toa sports scenario. A computer system in accordance with the principlesof the present invention may include an athlete wearable device 2502 fortracking the location of the athlete's general position on a field,track, court, etc. The wearable device 2502 may include GPS, localtriangulation system, etc. The wearable device 2502 may be worn in thewaist area of the athlete 2504 (e.g., on a belt) or otherwise near theathlete's center of gravity. The athlete's position and overall movementand momentum may be better estimated with the triangulation systemmounted near the center of gravity.

The computer system may track the location of the athlete as sheprogresses through a practice session or drill. The locations may beused to estimate the direction, speed, and momentum of the athlete 2504throughout the activities. An IMU, velocity sensor, speed sensor, motionsensor, etc. may be incorporated into the wearable device 2502 used tofurther assist in the prediction of the athlete's location and in aprediction how the athlete is moving and where the athlete is movingtowards. For example, a GPS system may track her position and an IMU maytrack her inertial movements. A short history of these measurements maybe used in a calculation of where the athlete is going to be in a shortperiod of time (e.g., 50 ms, 100 ms, 1 sec). It may be important topredict the athletes near-future position such that augmented realitycontent properly aligns with her position when the content is presented.This can reduce effects of latency in the process of generating,communicating, and presenting the content to the athlete.

A second athlete, possibly using the AR system as well, may be locationtracked like the athlete being trained. The second athlete may be ontrack to intersect with the athlete being trained and a prediction ofthe collision time, position and resulting movements may be made suchthat the AR content may be positioned properly in the AR headset(s). Forexample, the intersection may be a light engagement or a full tackle andthe AR content position within the headset may be shifted based on theinteraction or predicted interaction.

Head Position Tracking

The computer system may include a head or helmet tracking system 2506 toidentify the direction the helmet is facing. A helmet, for example, mayhave a compass system to detect the direction of the helmet. It may alsohave accelerometers, IMUs, motion sensors, g-force sensors, etc. thatmonitor the helmet's motions. IMUs, for example can be very fast butthey tend to drift and often require periodic calibration. By combininga relatively slow magnetometer with an IMU or other motion sensors thefast response IMU may be calibrated to the magnetometer output. This mayresult in a reliable and fast response time and data output indicativeof the helmet's position.

Motion or force sensors may also be worn on the neck of the athlete tomeasure the force of the various neck muscles as an indication of theperson's head position. The neck muscle data may further be combinedwith a compass and IMU type data from the helmet or other head wornmonitor. This may provide for another data source to calibrate the IMU,for example. It may also be used to confirm other head motiondetections.

The data from the helmet and/or neck may be fused in such a way as topredict accurate head position and tracking (e.g., as described above).The historical tracking of the helmet's position may be used to predicta future position of the helmet. By understanding the direction,location, and forces being applied to the helmet, a near-future position(e.g., 50 ms, 100 ms, 1 sec) into the future may be made. Thehead/helmet has a known or approximated mass so when the location andforces are known, or estimated, one can predict with good accuracy wherethe helmet may be in the near-future.

Another technology used to estimate an athlete's head/helmet positioncould be the use of a local LIDAR or other time of flight measurementsystem. Such a tracking system may be positioned near the athlete tomake the measurements. If the athlete is somewhat stationary, as with agoalie in hockey, a head tracking system may be set up on the goal ornear the goal. If the athlete is moving over a larger space, like aquarterback in football, the head tracking system may be held in aposition by a drone or wired control system such that it may move inconcert with the athlete.

Yet another technology used to estimate an athlete's head/helmetposition may be a time of flight or optical system mounted on the helmetand positioned to measure a distance to a known location(s). Forexample, the ground may be marked, either visibly or invisibly, withmany markers and an optical system may be arranged to view the ground totrack the helmet position with respect to the markers. Each marker in agiven area may be coded such that the tracking system knows where it isin more absolute terms as well as relative terms.

Eye Position Tracking

The athlete may wear glasses, a face shield, a helmet, or other headworn device 2506 and the device may include an optical imaging system todetect the direction of the user's eyes. With eye tracking, the ARcontent presentation may be more targeted and/or foveated.

Presentation of AR Content

With the data fusion described herein one may estimate where the athleteis, where her head is looking, and where her eyes may be focused. Onemay also predict into the near future where the athlete's head may be.With a near-future prediction of location and head position AR contentmay be positioned to appear in a head mounted see-through computerscreen worn by the athlete at the right time and place such that thecontent appears affixed in a geospatial position without having toanchor the content to a visible object or market.

An AR training system may include a gaming engine (e.g., a system thatproduces a virtual environment in which the athlete can be mapped) andmay be remote from the athlete. The remote system may communicatepresentation information to a processor in the athlete's head mountedsystem. The remote system may communicate with through a local network,wide area network, cell network, etc. With much of the processingoccurring remotely and involving wireless communications, latency can bea challenge. For example, a 50 or 100 ms delay between generation of amodel, communication of the model, and presentation of the model may beperceived by the user as jitter or misaligned content. This is one ofthe reasons that predicting the athletes near-future location, attitude,condition, etc. may be important in training.

There is disclosed above the use of AI for controlling or influencingvirtual assets WVR as well as transitioning from BVR to WVR. There isfurther disclosed above detecting trends, tendencies, etc. from ARflight data. The trends may be group trends or an individual's trends.There is further disclosed above training the pilot based on theobserved trends and tendencies. There is further disclosed aboveselecting individuals for specific missions based on their performance,trends, tendencies. There is further disclosed above providing guidanceor cues to a pilot.

With reference to FIG. 26 , there is disclosed an exemplary andnon-limiting embodiment of a training ecosystem that involves anevolution from books to AR flight and combat.

The inventors discovered new systems and methods for training, tracking,and predicting operational tendencies in various environments forpersonnel in the control of vehicles. The new systems provide for moreadvanced training, tracking of student performance, insight into studenttendencies, etc.

FIG. 26 provides a high-level illustration of an exemplary andnon-limiting embodiment of a training system 2600 in accordance with theprinciples of the present inventions. The system 2600 may be used toprovide different training tools at different skill levels whiletracking student performance for expanded, refined or more targetedtraining. As illustrated in FIG. 26 , a student may begin vehicletraining by learning in a classroom 2602. The classroom curriculum andstudent performance may be stored in a central repository 2614. As it isdetermined that the student is ready for the next training environment,the student may begin to train with a ground-based AR/VR system 2604.Again, with the curriculum and student performance being stored. Thenext step may use additional tools, such as a ground-based simulator2608 and then move to in-air training in a real airplane, or othervehicle on land, water, or air, using AR to simulate real in-flightsituations while flying 2610.

The training, tracking, prediction may continue after qualifying astudent to operate vehicles 2612. For example, data from operationalflights (e.g., sorties, combat situations, re-fueling) can be trackedand stored in the central repository 2614 for in-flight guidance andpost-flight analysis.

A suite of feedback tools 2622 may form part of system 2600 and mayimplement replay review and live play review of vehicles and objects invirtual or real airspaces as described above.

In accordance with an exemplary and non-obvious embodiment, the systemmay create and store a geospatial virtual environment comprising aplurality of entities each having one or more location attributes andcorresponding time attributes wherein at least one of the plurality ofentities is a virtual asset and wherein at least one of the plurality ofentities represents a real vehicle having a defined location within aphysical space having spatial coordinates that are mapped to the virtualenvironment. In real-time or near real-time, the system may then receiveupdated location information for one or more real vehicles. Suchreceived location may then be mapped into the virtual environment andused to update the location attributes of the appropriate vehicle.

In this manner, each entity may have an associated body of datacomprised of time stamped location data as the vehicle moves throughactual space and its corresponding location within the virtualenvironment to which it is mapped. Lastly, the stored data describingpart or all of the geospatial virtual environment may be displayed to anoperator a real vehicle mapped to and forming a part of the virtualenvironment.

In some instances, the real vehicle may be an airplane and the operatorof the plane may be the pilot. In some instances, the display device maybe adapted to display the mixed reality representation while the pilotis performing a task selected from the group consisting of performing atraining exercise, engaging in a battle and navigating a pathway. In thelatter instance, the pathway may be indicated by one or more virtualobjects displayed to guide the route of a plane.

In some embodiments, the display may present a visualization of aweapons system as deployed from either the real vehicle or a virtualasset. Likewise, the display may present a visualization of acountermeasure as deployed from either the real vehicle or a virtualasset. In some instances, the real vehicle, ay be a roller coaster orpart of a ground transportation system.

Generally, consistent with embodiments of the disclosure, programmodules may include routines, programs, components, data structures, andother types of structures that may perform particular tasks or that mayimplement particular abstract data types. Moreover, embodiments of thedisclosure may be practiced with other computer system configurations,including hand-held devices, general purpose graphics processor-basedsystems, multiprocessor systems, microprocessor-based or programmableconsumer electronics, application specific integrated circuit-basedelectronics, minicomputers, mainframe computers, and the like.Embodiments of the disclosure may also be practiced in distributedcomputing environments where tasks are performed by remote processingdevices that are linked through a communications network. In adistributed computing environment, program modules may be located inboth local and remote memory storage devices.

Furthermore, embodiments of the disclosure may be practiced in anelectrical circuit comprising discrete electronic elements, packaged orintegrated electronic chips containing logic gates, a circuit utilizinga microprocessor, or on a single chip containing electronic elements ormicroprocessors. Embodiments of the disclosure may also be practicedusing other technologies capable of performing logical operations suchas, for example, AND, OR, and NOT, including but not limited tomechanical, optical, fluidic, and quantum technologies. In addition,embodiments of the disclosure may be practiced within a general-purposecomputer or in any other circuits or systems.

Embodiments of the disclosure, for example, may be implemented as acomputer process (method), a computing system, or as an article ofmanufacture, such as a computer program product or computer readablemedia. The computer program product may be a computer storage mediareadable by a computer system and encoding a computer program ofinstructions for executing a computer process. The computer programproduct may also be a propagated signal on a carrier readable by acomputing system and encoding a computer program of instructions forexecuting a computer process. Accordingly, the present disclosure may beembodied in hardware and/or in software (including firmware, residentsoftware, micro-code, etc.). In other words, embodiments of the presentdisclosure may take the form of a computer program product on acomputer-usable or computer-readable storage medium havingcomputer-usable or computer-readable program code embodied in the mediumfor use by or in connection with an instruction execution system. Acomputer-usable or computer-readable medium may be any medium that cancontain, store, communicate, propagate, or transport the program for useby or in connection with the instruction execution system, apparatus, ordevice.

The computer-usable or computer-readable medium may be, for example butnot limited to, an electronic, magnetic, optical, electromagnetic,infrared, or semiconductor system, apparatus, device, or propagationmedium. More specific computer-readable medium examples (anon-exhaustive list), the computer-readable medium may include thefollowing: an electrical connection having one or more wires, a portablecomputer diskette, a random-access memory (RAM), a read-only memory(ROM), an erasable programmable read-only memory (EPROM or Flashmemory), an optical fiber, and a portable compact disc read-only memory(CD-ROM). Note that the computer-usable or computer-readable mediumcould even be paper or another suitable medium upon which the program isprinted, as the program can be electronically captured, via, forinstance, optical scanning of the paper or other medium, then compiled,interpreted, or otherwise processed in a suitable manner, if necessary,and then stored in a computer memory.

Embodiments of the present disclosure, for example, are described abovewith reference to block diagrams and/or operational illustrations ofmethods, systems, and computer program products according to embodimentsof the disclosure. The functions/acts noted in the blocks may occur outof the order as shown in any flowchart. For example, two blocks shown insuccession may in fact be executed substantially concurrently or theblocks may sometimes be executed in the reverse order, depending uponthe functionality/acts involved.

While certain embodiments of the disclosure have been described, otherembodiments may exist. Furthermore, although embodiments of the presentdisclosure have been described as being associated with data stored inmemory and other storage mediums, data can also be stored on or readfrom other types of computer-readable media, such as secondary storagedevices, like hard disks, solid state storage (e.g., USB drive), or aCD-ROM, a carrier wave from the Internet, or other forms of RAM or ROM.Further, the disclosed methods' stages may be modified in any manner,including by reordering stages and/or inserting or deleting stages,without departing from the disclosure.

Although the invention has been explained in relation to its preferredembodiment, it is to be understood that many other possiblemodifications and variations can be made without departing from thespirit and scope of the invention.

What is claimed is:
 1. A system, comprising: a memory in communicationwith a processor, the memory storing instructions that when executed bythe processor cause the processor to: (a) create and store a geospatialvirtual environment comprising a plurality of entities each having oneor more location attributes and corresponding time attributes wherein atleast one of the plurality of entities is a virtual asset and wherein atleast one of the plurality of entities represents a real vehicle havinga defined location within a physical space having spatial coordinatesthat are mapped to the virtual environment: (b) receive an updatedlocation of the real vehicle; (c) map the received location of the realvehicle to the geospatial virtual environment; (d) update the entity ofthe geospatial virtual environment corresponding to the real vehiclewith the mapped received location; and (e) output data comprising aportion of the geospatial virtual environment to a display deviceadapted to display to an operator of the real vehicle a mixed realityrepresentation of at least one virtual entity.
 2. The system of claim 1,wherein the real vehicle is an airplane.
 3. The system of claim 1,wherein the operator is a pilot of the real vehicle and the displaydevice is further adapted to display the mixed reality representationwhile the pilot is performing a task selected from the group consistingof performing a training exercise, engaging in a battle and navigating apathway.
 4. The system of claim 1, wherein the at least one of theplurality of entities represents a second real vehicle.
 5. The system ofclaim 1, wherein the mixed reality representation comprises avisualization of a weapons deployment from the real vehicle.
 6. Thesystem of claim 1, wherein the mixed reality representation comprises avisualization of a weapons deployment from the virtual asset.
 7. Thesystem of claim 1, wherein the mixed reality representation comprises avisualization of a countermeasure deployment from the real vehicle. 8.The system of claim 1, wherein the mixed reality representationcomprises a visualization of a countermeasure deployment from thevirtual asset.
 9. The system of claim 1, wherein the real vehicle is aroller coaster.
 10. The system of claim 1, wherein the vehicle comprisesa ground transportation system vehicle.